Guiding vehicles through vehicle maneuvers using machine learning models

ABSTRACT

In various examples, a trigger signal may be received that is indicative of a vehicle maneuver to be performed by a vehicle. A recommended vehicle trajectory for the vehicle maneuver may be determined in response to the trigger signal being received. To determine the recommended vehicle trajectory, sensor data may be received that represents a field of view of at least one sensor of the vehicle. A value of a control input and the sensor data may then be applied to a machine learning model(s) and the machine learning model(s) may compute output data that includes vehicle control data that represents the recommended vehicle trajectory for the vehicle through at least a portion of the vehicle maneuver. The vehicle control data may then be sent to a control component of the vehicle to cause the vehicle to be controlled according to the vehicle control data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/614,466, filed on Jan. 7, 2018. This application is a continuation ofU.S. patent application Ser. No. 16/241,005, titled “GUIDING VEHICLESTHROUGH VEHICLE MANEUVERS USING MACHINE LEARNING MODELS,” filed Jan. 7,2019. Each of these applications is incorporated herein by reference inits entirety.

BACKGROUND

For autonomous vehicles to operate correctly in all environments, theautonomous vehicles must be capable of safely performing vehiclemaneuvers, such as lane changes, lane splits, and turns. For example,for an autonomous vehicle to navigate through surface streets (e.g.,city streets, side streets, neighborhood streets, etc.) and on highways(e.g., multi-lane roads), the autonomous vehicle is required to maketurns, take lane splits, change lanes, and/or perform other vehiclemaneuvers.

Conventional approaches to performing such vehicle maneuvers require ahigh definition (HD) map that not only relies on accurate, pre-recordedmapping of roads, lanes of roads, and intersections within the roads toglobal coordinates, but also relies on mapping the locations for amultitude of static objects—such as street signs, traffic signals, lanemarkings, and the like—to global coordinates in order to performeffectively. In such approaches, the autonomous vehicle localizes itselfwith respect to the HD map and performs vehicle maneuvers by referencingthe location of the autonomous vehicle with respect to the HD map (e.g.,by comparing global coordinates of the autonomous vehicle to the globalcoordinates of the HD map).

These conventional approaches present several challenges and obstaclesthat not only make them less reliable, but also less capable ofuniversal implementation. For example, when the HD map is not updated orknown for a given location (e.g., a certain neighborhood, town, city,etc.), the autonomous vehicle may be incapable of safely operating insuch a location. As a result, the autonomous vehicle is limited totravel within locations that have been accurately mapped, and may notoperate safely or fully autonomously when traveling in locations notsupported by the HD map. Further, generating, maintaining, updating, andnavigating an HD map is computationally expensive, and requires largeamounts of processing power, energy, and bandwidth to enable safe andeffective operation of autonomous vehicles.

SUMMARY

Embodiments of the present disclosure relate to using machine learningmodels as guidance for vehicles (e.g., autonomous vehicles) inperforming vehicle maneuvers. More specifically, systems and methods aredisclosed that use machine learning model(s) to provide guidance forvehicles when performing lane changes, lane splits, turns, and/or othervehicle maneuvers.

In contrast to conventional systems, such as those described above, thecurrent system uses a machine learning model(s) that computes vehiclecontrol data representative of a trajectory and/or controls forfollowing the trajectory for an autonomous vehicle when performing avehicle maneuver. The machine learning model(s) may perform thecomputations based on one or more of sensor data generated by sensor(s)of the autonomous vehicle (e.g., camera(s), RADAR sensors, LIDARsensors, etc.), control inputs representative of a type of and progressthrough a vehicle maneuver (e.g., right turn, left lane change, rightlane split, etc.), low-resolution map data (e.g., two-dimensionalrepresentations of intersections, basic geometry of the road and/orintersection, etc.), and/or vehicle status data (e.g., a current speedof the autonomous vehicle).

As a result, the autonomous vehicle is able to navigate through lanechanges, lane splits, turns, and other vehicle maneuvers by relyingprimarily on computer vision rather than localization using an HD map(e.g., persistently computing a location of the vehicle with respect tostatic objects, lanes, intersections, etc. as represented on the HDmap). By removing the reliance on an HD map, the autonomous vehicle isable to perform lane changes, lane splits, turns, and/or other vehiclemaneuvers in any location, regardless of whether an HD map is availablefor that location. In addition, by using a machine learning model(s) andsensor data generated by the autonomous vehicle, vehicle maneuversperformed according to the present disclosure may be lesscomputationally expensive than conventional approaches, and require lessprocessing power, energy consumption, and bandwidth.

BRIEF DESCRIPTION OF THE DRAWINGS

The present systems and methods for using machine learning models asguidance for vehicles in performing vehicle maneuvers is described indetail below with reference to the attached drawing figures, wherein:

FIG. 1A is a data flow diagram illustrating an example process forperforming a vehicle maneuver, in accordance with some embodiments ofthe present disclosure;

FIG. 1B is an illustration of an example machine learning model(s), inaccordance with some embodiments of the present disclosure;

FIG. 2 is an illustration of an example vehicle maneuver and acorresponding chart of control input values for a machine learningmodel(s) over time for the vehicle maneuver, in accordance with someembodiments of the present disclosure;

FIG. 3 is an illustration of an example vehicle maneuver and exampletraining data for a machine learning model(s), in accordance with someembodiments of the present disclosure;

FIG. 4 is a flow diagram showing a method for performing a vehiclemaneuver, in accordance with some embodiments of the present disclosure;

FIG. 5A is a data flow diagram illustrating an example process forperforming a vehicle maneuver, in accordance with some embodiments ofthe present disclosure;

FIG. 5B is an illustration of another example machine learning model(s),in accordance with some embodiments of the present disclosure;

FIG. 5C is an illustration of another example machine learning model(s),in accordance with some embodiments of the present disclosure;

FIG. 6 is a flow diagram showing a method for performing another vehiclemaneuver, in accordance with some embodiments of the present disclosure;

FIG. 7 is a flow diagram showing a method for performing another vehiclemaneuver, in accordance with some embodiments of the present disclosure;

FIG. 8A is an illustration of an example autonomous vehicle, inaccordance with some embodiments of the present disclosure;

FIG. 8B is an example of camera locations and fields of view for theexample autonomous vehicle of FIG. 8A, in accordance with someembodiments of the present disclosure;

FIG. 8C is a block diagram of an example system architecture for theexample autonomous vehicle of FIG. 8A, in accordance with someembodiments of the present disclosure; and

FIG. 8D is a system diagram for communication between cloud-basedserver(s) and the example autonomous vehicle of FIG. 8A, in accordancewith some embodiments of the present disclosure.

DETAILED DESCRIPTION

Systems and methods are disclosed related using machine learning modelsas guidance for vehicles in performing vehicle maneuvers. The presentdisclosure may be described with respect to an example autonomousvehicle 140 (alternatively referred to herein as “vehicle 140” or“autonomous vehicle 140”), an example of which is described in moredetail herein with respect to FIGS. 8A-8D. In addition, although thevehicle maneuvers described herein include primarily lane changes, lanesplits, and turns, the present disclosure is not intended to be limitedto only these vehicle maneuvers. For example, operations performed foror in the course of parking, reversing, and/or other vehicle maneuversmay also benefit from the methods described herein. In addition,although the description in the present disclosure separates lanechanges from lane splits and turns, this is not intended to be limiting.For example, features and functionality described herein with respect tolane changes may also be applicable to lane splits and/or turns. In thealternative, features and functionality described herein with respect tolane splits and/or turns may also be applicable to lane changes.

Lane Changes

As described above, conventional systems rely on an HD map forperforming lane changes. However, by relying on an HD map, an autonomousvehicle must first have accurate location information (e.g., from a GPSsensor) and must also have an accurate HD map. As a result, when an HDmap does not exist and/or when the location information is updatingslowly and/or is inaccurate, lane changing may not be performed safely.The autonomous vehicle may then not perform the lane change (e.g., mayimplement a safety protocol to abort the lane change), perform anerratic lane change (e.g., because the location information isinaccurate, so the localization of the vehicle with respect to the lanemarkings may suffer as a result), and/or require hand-off of thecontrols to a human driver. None of these options are particularlydesirable for a passenger of the autonomous vehicle.

The present systems provide for an autonomous vehicle 140 that may makelane changes without relying on an HD map and/or location information.Instead, the autonomous vehicle may rely on computer vision using sensordata generated from one or more sensors of the vehicle (e.g., cameras,LIDAR sensors, RADAR sensors, etc.). As a result, the changing of thelanes is perception based, and thus may avoid reliance on potentiallyunavailable, obsolete, and/or otherwise inaccurate information (e.g., HDmaps, location information, etc.).

For example, a trigger signal may be received (e.g., left lane change,right lane change) in response to an input (e.g., a user input to a turnsignal or blinker switch, an input from a control system (e.g., inresponse to a determination made by a planning layer of an autonomousdriving software stack), and/or otherwise in response to a command fromthe system. In response to (or based on) the trigger signal, a leftcontrol operand (for changing lanes left) or a right control operand(for changing lanes right) may be updated to an initial value (e.g., 1,100%, etc.) to indicate the beginning of a lane change procedure ormode. The left control operand and/or right control operand may be aninput into a machine learning model(s) (e.g., a neural network, such asa convolutional neural network). In addition, sensor data from one ormore sensors of the vehicle (e.g., cameras, LIDAR sensors, RADARsensors, etc.) may be provided as input to the machine learningmodel(s).

The machine learning model(s) may output vehicle trajectory points(e.g., (x, y) coordinates) representative of a recommended vehicletrajectory and/or proposed controls for a vehicle control unit forcontrolling the vehicle according to the recommended vehicle trajectory.During performance of the lane change, the machine learning model(s) maycontinue to receive the sensor data and may calculate the status of thelane change as represented by progress data (e.g., representative of theprogress of the vehicle through the lane change) output by the machinelearning model(s). This output, in some examples after going throughcontrol logic to determine a control input update based on the progressdata, may be fed back to the left control operand or the right controloperand (e.g., dependent on the direction of the lane change), and theupdated value for the left control operand or the right control operandmay be provided to the machine learning model(s) in a next iteration.This process may continue until the lane change is completed.

In some examples, a lane change may be discretized into a series ofstages. For example, a lane change may include two stages, three stages,four stages, and/or another number of stages. Some lane changes mayinclude a different number of stages than other lane changes. Forexample, a first lane change may include two stages, while a second lanechange may include three stages, a third lane change may include fivestages, and so on. In some examples, a determination may be made (e.g.,when the trigger signal is received) of the type of lane change and/orvariables or other information in the environment that effect the lanechange (e.g., a left lane change, a right lane change, a left lanechange on a curve, a right lane change in between two vehicles, etc.).Depending on the determination of the type of lane change and/or thevariables or other information in the environment, a determination maybe made as to how many stages should be included in the lane change.

For example, in a non-limiting example, for a left lane change, whenanother object is currently next to the vehicle in the destination lane,a determination may be made that three stages are necessary (e.g., afirst stage to accelerate or decelerate such that the object is nolonger to the left of the vehicle, a second stage from the end of thefirst stage until the vehicle is approximately midway between theoriginal lane and the destination lane, and third stage between the endof the second stage and when the vehicle is in the destination lane).

In another non-limiting example, the lane change may include at leasttwo stages—a first stage from a beginning of the lane change to themiddle of the lane change (e.g., approximately when the centerline ofthe vehicle lines up with the lane markings separating the original lanefrom the destination lane, or when the vehicle is roughly equidistant tothe outer lane markings of the original lane and the destination lane)and a second stage from the middle of the lane change until the end (oruntil a progress threshold that represents an end of the lane change).In such examples, the left control operand or the right control operandmay be updated based on progress thresholds representative of athreshold amount of the lane change that the vehicle has progressedthrough (e.g., for a right lane change, right control operand may beapproximately 1.0 throughout the first stage, approximately 0.5throughout the second stage, and approximately 0.0 after the lane changeis completed and/or after a progress threshold representative of the endis reached). The stage change may take place when the lane changeprogress exceeds the progress thresholds (e.g., between 0%-50% complete,remain in stage one, at 50% complete, enter stage two, at 80% complete,exit lane change procedure or mode, and begin a lane keeping procedureor mode in the destination lane).

By discretizing a lane change into a series of multiple stages, thesystem may be more able to clearly determine when a lane change beginsand ends and may be more able to clearly distinguish between sensor datathat may appear similar during different portions of the same lanechange. For example, at 48% complete and 52% complete, the sensor datamay be relatively similar, so by starting stage two of the lane changeat or approximately at 50%, the system may understand that performingthe second stage of the lane change focuses on entering and aligning thevehicle with the destination lane (e.g., entering a lane keepingprocedure or mode). In addition, by only changing the left controloperand or right control operand in response to thresholds being met,the noisy outputs of the machine learning model(s) (e.g., 70% completed,then 68% completed, then 72% completed, etc.) may be filtered such thatthe resultant vehicle trajectories do not suffer from the noise of theoutputs (e.g., do not result in sporadic or erratic vehicletrajectories).

In some examples, lane change parameters (e.g., vehicle maneuverparameters) may also be input into the machine learning model(s), suchas a lane change length (e.g., a distance that the vehicle should coverduring the lane change), an amount of time for completing the lanechange, a velocity or speed of the lane change, and/or other lane changeparameters. The lane change parameters may be used by the machinelearning model(s) to determine the vehicle trajectory points forgenerating the recommended vehicle trajectory.

Now referring to FIG. 1A, FIG. 1A is a data flow diagram illustrating anexample process 100 for performing a vehicle maneuver, in accordancewith some embodiments of the present disclosure. The vehicle maneuverdescribed with respect to FIG. 1A is a lane change, however, asdescribed herein, this is for example purposes only and is not intendedto be limiting.

The vehicle maneuver may begin when a trigger signal (e.g., from asignal generator 102) is received. Any number of inputs 104, includingbut not limited to those illustrated in FIG. 1A, may be input into amachine learning model(s) 118. The machine learning model(s) 118 maygenerate or compute any number of outputs 120, including but not limitedto those illustrated in FIG. 1A. At least one of the outputs 120 may befed back into the inputs 104, as indicated by feedback loop 126 (e.g.,the outputs 120 may be used to update one or more of the inputs 104 fora next iteration of the machine learning model(s) 118. At least one ofthe outputs 120 may be transmitted or sent to a control component(s) 128of the autonomous vehicle 140. The control component(s) 128 may then usethe output(s) 120 (or information generated from the output(s) 120) tocontrol the vehicle 140 according to the output(s) 120.

The trigger signal from the signal generator 102 may include datarepresentative of the type of vehicle maneuver to be completed by thevehicle 140. For example, the trigger signal may include datarepresenting or otherwise indicative of a right lane change (e.g.,change lanes from current lane to adjacent lane(s) to the right of thevehicle 140), a left lane change (e.g., change lanes from current laneto adjacent lane(s) to the left of the vehicle 140), and/or othervehicle maneuver, such as a turn or lane split, as described in moredetail below with respect to FIGS. 5A-5C. In some examples, the triggersignal may be generated by the signal generator 102 using pulse widthmodulation (PWM), or another signal generation method, and may beanalyzed by a component(s) of the vehicle 140 to determine the type ofvehicle maneuver represented by the trigger signal.

The trigger signal may be received in response to an input received bythe vehicle 140 and/or in response to a decision made by the vehicle140. For example, an operator of the vehicle 140 may provide an inputvia a turn signal, a keyboard, a human machine interface (HMI) display,voice input, and/or another input type. The input may be representativeof a specific vehicle maneuver and, as a result, the trigger signalrepresenting the specific vehicle maneuver may be generated. As anotherexample, the vehicle 140 (e.g., one or more components or features of aplanning layer of an autonomous vehicle software stack) may determinethat a specific vehicle maneuver is required or desired and, as aresult, may generate the trigger signal representing the specificvehicle maneuver. In such an example, the vehicle 140 may use an HD map,navigational guidance, and/or another information source to determinethat the specific vehicle maneuver is required or desired (e.g., toincrease safety). In some examples, the trigger signal may include orcorrespond to a guidance signal generated by a GNSS application (e.g., aGPS application).

The inputs 104 may include a left control operand 106, a right controloperand 108, sensor information 110, status information 112, a lanechange parameter(s) 114, map information 116, and/or other inputs.Values of the left control operand 106 and the right control operand 108may be determined based on the trigger signal and/or feedback from theoutputs 120 of the machine learning model(s) 118. For example, when thetrigger signal is representative of a left lane change, the value of theleft control operand 106 may be set, updated, and/or changed based onthe trigger signal. As another example, when the trigger signal isrepresentative of a right lane change, the value of the right controloperand 108 may be set, updated, and/or changed based on the triggersignal. In any example, the machine learning model(s) 118 may use theleft control operand 106 and/or the right control operand 108 incomputing vehicle control information 124, lane change progress 122,and/or other outputs 120.

In some examples, for a left lane change, in response to receiving thetrigger signal, the value of the left control operand 106 may be set,updated, and/or changed to 1, 100%, and/or another value indicating astart of the left lane change. As the lane change progresses, the valueof the left control operand 106 may be updated to represent the progressof the vehicle 140 through the lane change. For example, based on one ofthe outputs 120, such as the lane change progress 122, the value of theleft control operand 106 may be set, updated, and/or changed. In someexamples, the value of the left control operand 106 may be set, updated,and/or changed incrementally at each iteration of using the machinelearning model(s) 118 for the vehicle maneuver, until the left lanechange has been completed (e.g., 0.9 or 90% when ten percent of lanechange is completed, 0.45 or 45% when fifty-five percent of the lanechange is completed, etc.). In other examples, the value of the leftcontrol operand 106 may be set, updated, and/or changed in response tocertain progress thresholds being met, as described herein at least withrespect to FIG. 2 (e.g., the value may be set to 1 when the progressthrough the lane change is less than fifty percent completed, 0.5 whenthe progress through the lane change is between fifty percent and a nextthreshold, 0 when the progress through the lane change is eighty percentor more completed, etc.). In such examples, the progress thresholds maybe used to account for potentially noisy outputs 120 of the machinelearning model(s) 118.

Similarly, for a right lane change, in response to receiving the triggersignal, the value of the right control operand 108 may be set, updated,and/or changed to 1, 100%, and/or another value indicating a start ofthe right lane change. As the lane change progresses, the value of theright control operand 108 may be updated to represent the progress ofthe vehicle 140 through the lane change. For example, based on one ofthe outputs 120, such as the lane change progress 122, the value of theright control operand 108 may be set, updated, and/or changed. In someexamples, the value of the right control operand 108 may be set,updated, and/or changed at each iteration of the machine learningmodel(s) 118, until the right lane change has been completed. In otherexamples, the value of the right control operand 108 may be set,updated, and/or changed in response to certain progress thresholds beingmet, as described herein at least with respect to FIG. 2.

Another input 104 that may be applied to the machine learning model(s)118 is sensor information 110. The sensor information 110 may includesensor data from any of the sensors of the vehicle 140 (and/or othervehicles, in some examples). For example, with reference to FIGS. 8A-8C,the sensor information 110 may include the sensor data generated by, forexample and without limitation, global navigation satellite systems(GNSS) sensor(s) 858 (e.g., Global Positioning System sensor(s)), RADARsensor(s) 860, ultrasonic sensor(s) 862, LIDAR sensor(s) 864, inertialmeasurement unit (IMU) sensor(s) 866 (e.g., accelerometer(s),gyroscope(s), magnetic compass(es), magnetometer(s), etc.),microphone(s) 896, stereo camera(s) 868, wide-view camera(s) 870 (e.g.,fisheye cameras), infrared camera(s) 872, surround camera(s) 874 (e.g.,360 degree cameras), long-range and/or mid-range camera(s) 898, speedsensor(s) 844 (e.g., for measuring the speed of the vehicle 140),vibration sensor(s) 842, steering sensor(s) 840, brake sensor(s) (e.g.,as part of the brake sensor system 846), and/or other sensor types.

In some examples, the sensor information 110 may include the sensor datagenerated by a forward-facing camera, such as a wide-view camera 870, asurround camera 874, a stereo camera 868, and/or a long-range ormid-range camera 898. This sensor data may be useful for computer visionand/or perception when navigating a lane change because a forward-facingcamera may include a field of view (e.g., the field of view of theforward-facing stereo camera 868 and/or the wide-view camera 870 of FIG.8B) that includes both a current lane of travel of the vehicle 140 andadjacent lane(s) of travel of the vehicle 140. In some examples, morethan one camera or other sensor may be used to incorporate multiplefields of view (e.g., the fields of view of the long-range cameras 898,the forward-facing stereo camera 868, and/or the forward facingwide-view camera 870 of FIG. 8B).

In any example, the sensor information 110 may include image datarepresenting an image(s), image data representing a video (e.g.,snapshots of video), and/or sensor data representing fields of view ofsensors (e.g., LIDAR sensor(s) 864, RADAR sensor(s) 860, etc.). Thesensor information 110 may be input into the machine learning model(s)118 and used by the machine learning model(s) 118 to compute the outputs120, such as the lane change progress 122 and/or the vehicle controlinformation 124.

The inputs 104 may include status information 112. The statusinformation 112 may include data representative of the status of thevehicle 140, such as speed, velocity, acceleration, deceleration,orientation, and/or other status information. This data may be capturedby and/or received from one or more of the sensors of the vehicle 140,such as one or more of the IMU sensor(s) 866, speed sensor(s) 844,steering sensor(s) 840, vibration sensor(s) 842, and/or one or moresensors of the brake sensor system 846, propulsion system 850, and/orsteering system 854.

The status information 112 may be used by the machine learning model(s)118 to compute the outputs 120, such as the lane change progress 122and/or the vehicle control information 124. For example, when thevehicle control information 124 includes a recommended vehicletrajectory, or control data for controlling the vehicle 140 according toa recommended vehicle trajectory, the status information 112 (e.g.,speed, orientation, etc.) may be valuable to the machine learningmodel(s) 118 in performing such computations. In other words, a vehicletrajectory that is representative of recommended future (x, y)coordinates of the vehicle 140 may benefit from knowledge of the currentspeed, orientation, and/or other status information 112 of the vehicle140.

In some examples, the inputs 104 may further include the lane changeparameter(s) 114 (e.g., vehicle maneuver parameters). The lane changeparameter(s) 114 may include a length of a lane change (e.g., a minimum,maximum, and/or range of distance the vehicle 140 may travel, such as ina forward direction, when performing the lane change), a duration of thelane change (e.g., an amount of time within which the lane change shouldbe completed), a speed or velocity of the lane change, a tempo factor(e.g., a measure, such as on a scale of values (e.g., 1-10), that isindicative of the speed and/or tempo of the lane change), and/or anotherlane change parameter 114.

The lane change parameter(s) 114 may be used by the machine learningmodel(s) 118 to compute the outputs 120, such as the lane changeprogress 122 and/or the vehicle control information 124. For example,when computing a recommended vehicle trajectory, the machine learningmodel(s) 118 may use the lane change parameter(s) 114 because the lanechange parameter(s) 114 may indicate where along the recommended vehicletrajectory the lane change should end (e.g., based at least in part onthe length of the lane change), how quickly the lane change should becompleted (e.g., based at least in part on the duration of the lanechange), and/or a combination thereof (e.g., based at least in part onthe tempo factor). In addition, the lane change progress 122 may beaffected by the lane change parameter(s) 114 (e.g., the lane changeprogress 122 will increase more quickly for shorter lane changes,shorter durations, etc.).

Further, in some examples, the inputs 104 may include map information116. The map information 116, as described in more detail herein withrespect to FIGS. 5A-5C, may be used by the machine learning model(s) 118to generate the outputs 120, such as the lane change progress 122 and/orthe vehicle control information 124. For example, the map information116 may include low-resolution map data (e.g., screenshots of a 2D mapapplication with or without guidance). This low-resolution map data mayinclude a basic geometry of the road and/or intersections, such aswithout additional information such as lane markings, number of lanes,locations of sidewalks, street lights, stop signs, etc. In other words,in contrast with the map data representing an HD map (e.g., the HD map822 and/or the HD maps described herein and relied upon by conventionalsystems), the map information 116 may be less data intense, and usedonly as an additional data point by the machine learning model(s) 118when computing the outputs 120.

The map information 116, in some examples, may include a screenshot oran image (or data representative thereof) that depicts a current lane ofthe vehicle 140, a destination lane of the vehicle 140, the vehicle 140itself, and/or a representation of the path for the vehicle 140 to takethrough the lane change (e.g., similar to path 546 of FIG. 5B). In someexamples, the map information 116 may be similar to that of the mapinformation 514A of FIG. 5B and/or the map information 514B of FIG. 5C,except rather than illustrating an intersection 536 and/or the path 546through the intersection, the map information 116 may include dataindicative of a first lane, a second lane, and/or a path for navigatingbetween them.

In examples where the map information 116 is one of the inputs 104, andthe map information 116 includes the path for navigating from one laneto another, the left control operand 106 and/or the right controloperand 108 may be used differently or not at all. For example, the leftcontrol operand 106 and/or the right control operand 108 may onlyinclude a start value (e.g., 1, 100%, etc.) for the start of the lanechange and an end value (e.g., 0, 0%, etc.) for the end of the lanechange. In other examples, the left control operand 106 and/or the rightcontrol operand 108 may not be needed. In either example, the leftcontrol operand 106 and/or the right control operand 108 may bedifferent because the vehicle's location with respect to the path mayindicate the progress of the vehicle through the lane change, so themachine learning model(s) 118 may not require the left control operand106 and/or the right control operand 108 when computing the lane changeprogress 122 (e.g., the vehicle 140 may rely on the path to make thiscomputation instead).

The machine learning model(s) 118 may use the inputs 104 to compute theoutputs 120. Although examples are described herein with respect tousing neural networks, and specifically convolutional neural networks,as the machine learning model(s) 118 (e.g., with respect to FIGS. 1B and5B-5C), this is not intended to be limiting. For example, and withoutlimitation, the machine learning model(s) 118 may include any type ofmachine learning model, such as a machine learning model(s) using linearregression, logistic regression, decision trees, support vector machines(SVM), Naïve Bayes, k-nearest neighbor (Knn), K means clustering, randomforest, dimensionality reduction algorithms, gradient boostingalgorithms, neural networks (e.g., auto-encoders, convolutional,recurrent, perceptrons, long/short term memory/LSTM, Hopfield,Boltzmann, deep belief, deconvolutional, generative adversarial, liquidstate machine, etc.), and/or other types of machine learning models.

The outputs 120 may include lane change progress 122, vehicle controlinformation 124, and/or other output types. The lane change progress 122may be representative of the progress of the vehicle 140 through thelane change. As described herein, the lane change progress 122 may befed back to the inputs 104, and specifically may be fed back to the leftcontrol operand 106 and/or the right control operand 108 for setting,updating, and/or changing the left control operand 106 and/or rightcontrol operand 108. In some examples, the machine learning model(s) 118may output data representing the lane change progress 122, and logic(e.g., control logic 136 of FIG. 1B) may be used to determine thesetting, updating, and/or changing of the left control operand 106and/or right control operand 108 based on the data.

The outputs 120 may further include the vehicle control information 124.The vehicle control information 124 may include a recommended vehicletrajectory and/or vehicle control data for controlling the vehicleaccording to the recommended vehicle trajectory. The outputs 120 mayinclude one or more points (e.g., (x, y) coordinates) along therecommended vehicle trajectory and/or may include the recommendedvehicle trajectory (e.g., a trajectory extrapolated over each of thepoints). In some examples, the recommended vehicle trajectory may berepresented as a radius of the vehicle maneuver, while in otherexamples, the recommended vehicle trajectory may be represented as aninverse radius of the vehicle maneuver. The inverse radius may be usedin some examples to prevent the recommended vehicle trajectory (or apoint thereof) from being computed as an infinite value (e.g., reachingsingularity).

In examples where the vehicle control information 124 includes therecommended vehicle trajectory, the recommended vehicle trajectory (ordata representative thereof) may be sent or transmitted to the controlcomponent(s) 128 of the vehicle 140 (e.g., to a control layer of theautonomous driving software), and the control component(s) 128 maydetermine the controls required to control the vehicle 140 according tothe recommended vehicle trajectory. For example, the controlcomponent(s) 128 may send one or more controls to one or more actuators(e.g., actuators controlled by an actuation layer of the autonomousdriving software stack). The actuators may include one or morecomponents or features of the brake sensor system 846, the propulsionsystem 850, the steering system 854, and/or other systems. The vehicle140 may then be controlled according to the recommended vehicletrajectory of the vehicle control information 124 output by the machinelearning model(s) 118. By only outputting the recommended vehicletrajectory and not the controls themselves, the process 100 allowdifferent vehicle manufacturers to determine their own controls andactuations for controlling the vehicle 140 according to the recommendedvehicle trajectory.

In other examples, the vehicle control information 124 may include thecontrols for controlling the vehicle according to the recommendedvehicle trajectory. In such examples, the machine learning model(s) 118may be implemented at the control layer of the autonomous drivingsoftware stack, and the vehicle control information 124 may be used tocontrol the vehicle 140 (e.g., cause actuation of one or more actuatorsof the vehicle by the actuation layer of the autonomous driving softwarestack).

Now referring to FIG. 1B, FIG. 1B is an illustration of an examplemachine learning model(s) 118A, in accordance with some embodiments ofthe present disclosure. The machine learning model(s) 118A of FIG. 1Bmay be one example of a machine learning model(s) 118 that may be usedin the process 100. However, the machine learning model(s) 118A of FIG.1B is not intended to be limiting, and the machine learning model(s) 118may include additional and/or different machine learning models than themachine learning model(s) 118A of FIG. 1B. The machine learning model(s)118A may include a convolutional neural network and thus mayalternatively be referred to herein as convolutional neural network 118Aor convolutional network 118A.

The convolutional network 118A may use inputs 104 that include the leftcontrol operand 106, the right control operand 108, the lane changeparameter(s) 114, the sensor information 110, and/or other input types.The convolutional network 118A may use sensor information 110A-110Cwhich may include image data generated by one or more cameras (e.g., oneor more of the cameras described herein with respect to FIGS. 8A-8C).For example, the sensor information 110A-110C may include image datarepresentative of a field of view of the camera(s). More specifically,the sensor information 110A-110C may include individual images generatedby the camera(s), where image data representative of one or more of theindividual images is input into the convolutional network 118A at eachiteration of the convolutional network 118A.

The sensor information 110 may be input into a convolutional stream 130of the convolutional network 118A. The convolutional stream 130 mayinclude any number of layers 132, such as the layers 132A-132C. One ormore of the layers 132 may include an input layer. The input layer mayhold values associated with the sensor information 110. For example,when the sensor information 110 is an image(s), the input layer may holdvalues representative of the raw pixel values of the image(s) as avolume (e.g., a width, a height, and color channels (e.g., RGB), such as32×32×3).

One or more layers 132 may include convolutional layers. Theconvolutional layers may compute the output of neurons that areconnected to local regions in an input layer (e.g., the input layer),each neuron computing a dot product between their weights and a smallregion they are connected to in the input volume. A result of theconvolutional layers may be another volume, with one of the dimensionsbased on the number of filters applied (e.g., the width, the height, andthe number of filters, such as 32×32×12, if 12 were the number offilters).

One or more of the layers 132 may include a rectified linear unit (ReLU)layer. The ReLU layer(s) may apply an elementwise activation function,such as the max (0, x), thresholding at zero, for example. The resultingvolume of a ReLU layer may be the same as the volume of the input of theReLU layer.

One or more of the layers 132 may include a pooling layer. The poolinglayer may perform a down sampling operation along the spatial dimensions(e.g., the height and the width), which may result in a smaller volumethan the input of the pooling layer (e.g., 16×16×12 from the 32×32×12input volume).

One or more of the layers 132 may include a fully connected layer. Eachneuron in the fully connected layer(s) may be connected to each of theneurons in the previous volume. The fully connected layer may computeclass scores, and the resulting volume may be 1×1×number of classes. Insome examples, the convolutional stream(s) 130 may include a fullyconnected layer, while in other examples, the fully connected layer 134of the convolutional network 118A may be the fully connected layer forthe convolutional stream(s) 130.

Although input layers, convolutional layers, pooling layers, ReLUlayers, and fully connected layers are discussed herein with respect tothe convolutional stream(s) 130, this is not intended to be limiting.For example, additional or alternative layers 132 may be used in theconvolutional stream(s) 130, such as normalization layers, SoftMaxlayers, and/or other layer types.

Different orders and numbers of the layers 132 of the convolutionalnetwork 118A may be used depending on the embodiment. For example, for afirst vehicle, there may be a first order and number of layers 232,whereas there may be a different order and number of layers 232 for asecond vehicle; for a first camera there may be a different order andnumber of layers 232 than the order and number of layers for a secondcamera. In other words, the order and number of layers 232 of theconvolutional network 118A and/or the convolutional stream 130 is notlimited to any one architecture.

In addition, some of the layers 232 may include parameters (e.g.,weights), such as the convolutional layers and the fully connectedlayers, while others may not, such as the ReLU layers and poolinglayers. In some examples, the parameters may be learned by theconvolutional stream 130 and/or the machine learning model(s) 118Aduring training. Further, some of the layers 232 may include additionalhyper-parameters (e.g., learning rate, stride, epochs, etc.), such asthe convolutional layers, the fully connected layers, and the poolinglayers, while other layers 232 may not, such as the ReLU layers. Theparameters and hyper-parameters are not to be limited and may differdepending on the embodiment.

The output of the convolutional stream(s) 130 may be input to a fullyconnected layer(s) 134 of the convolutional network 118A. In addition tothe output of the convolutional stream(s) 130, the lane changeparameter(s), the left control operand 106, the right control operand108, and/or one or more other inputs 104 may be input to the fullyconnected layer(s) 134.

The outputs 120 of the convolutional network 118A may include the lanechange progress 122, the vehicle control information 124, and/or otheroutput types. The vehicle control information 124, as described herein,may include a recommended vehicle trajectory 144 and/or controls forfollowing the recommended vehicle trajectory 144 (e.g., for controllingthe vehicle 140 according to the recommended vehicle trajectory 144,such as steering angle, acceleration, deceleration, etc.). The vehiclecontrol information 124 may include, in some examples, a trajectorypoint(s) 142 (e.g., as represented by (x, y) coordinates) along therecommended vehicle trajectory 144. In some examples, only a singletrajectory point 142 (e.g., the next trajectory point for the vehicle140 in the sequence of discretized trajectory steps) may be output bythe machine learning model(s) 118A. In other examples, more than onetrajectory point 142 may be output. As another example, an entiretrajectory may be output, which may be extrapolated from two or moretrajectory points 142. In any example, the recommended vehicletrajectory 144 may be output as a radius of the recommended vehicletrajectory 144, or may be output as an inverse radius of the recommendedvehicle trajectory 144, as described herein. The recommended vehicletrajectory 144 and/or the trajectory point(s) 142 thereon, may be usedby the control component(s) 128 to control the vehicle 140 from a firstlane 146A to a second lane 146B to execute a lane change (e.g., a leftlane change as indicated in FIG. 1B).

With reference to the lane change progress 122, the lane change progress122 may be output as data representative of the lane change progress122, and the data may be analyzed to determine the lane change progress122. In other words, the output of the machine learning model(s) 118 maynot be the value of the lane change progress 122, but may be a valuethat correlates to the lane change progress 122. For example, a lookuptable may be used to determine the lane change progress 122 based on theoutput of the machine learning model(s) 118.

As described herein, the lane change progress 122 may be fed back to theleft control operand 106 and/or the right control operand 108 to updatethe values. In some examples, the control logic 136 may use the lanechange progress 122 output by the convolutional network 118A todetermine a setting, and update, and/or a change to the left controloperand 106 and/or a right control operand 108. In some examples, theleft control operand 106 and/or the right control operand 108 may beupdated in sync with the lane change progress 122. For example, withrespect to a right lane change, when the lane change progress 122 isindicative of the right lane change being a certain percentage complete(e.g., 70%), the control logic 136 may determine that the right controloperand 108 should be set, updated, and/or changed to a specific value(e.g., when the lane change progress 122 is 70%, the specific value maybe 0.3, indicating that there is 30% left of the right lane change). Assuch, for each value of the lane change progress 122, there may be acorresponding value to set, update, and/or change the left controloperand 106 and/or the right control operand 108 to. In some examples,the left control operand 106 may include positive values (e.g., 0-1,0%-100%, etc.), and the right control operand 108 may include negativevalues (e.g., −1-0, −100%-0%, etc.), or vice versa. In such examples,the left control operand 106 and the right control operand 108 may be asingle control (e.g., a universal control), and may range from −1 to 1,or −100% to 100%, and/or the like.

In other examples, the left control operand 106 and/or the right controloperand 108 may be updated only once certain progress thresholds are met(e.g., thresholds indicative of the progress of the vehicle 140 throughthe vehicle maneuver, such a lane change). In such examples, the controllogic 136 may only update the left control operand 106 and/or the rightcontrol operand 108 when the progress thresholds are met. An example ofusing progress thresholds is illustrated with respect to FIG. 2.

FIG. 2 is an illustration of an example vehicle maneuver 200 and acorresponding chart 218 of control input values for a machine learningmodel(s) over time for the vehicle maneuver, in accordance with someembodiments of the present disclosure. In FIG. 2, the vehicle 140 ismaking a left lane change from a first lane 204A to a second lane 204Bto the left of the first lane 204A. An example trajectory of the vehicle140 through the lane change is indicated by the trajectory line 206.With respect to the setting, updating, and/or changing the value for theleft control operand 106 in response to progress thresholds and/orstages of the left lane change, the control logic 136 of FIG. 1B may beused.

In this example, the vehicle 140 performs the left lane change in twostages, a first stage 208 and a second stage 210 (in other examples, anynumber of stages may be used). The first stage 208 may be from thebeginning of the lane change (e.g., when the trigger signal is received)until the middle of the lane change (e.g., approximately when thecenterline of the vehicle lines up with the lane markings separating theoriginal lane from the destination lane, or when the vehicle is roughlyequidistant to the outer lane markings of the original lane and thedestination lane). The beginning of the lane change may have a firstprogress threshold 212, T0, indicating that approximately 0% of the lanechange has been completed. At the first progress threshold 212, T0, thevalue of the left control operand 106 may be approximately 1.0, 100%, oranother value indicating that the lane change is beginning, as indicatedby the chart 218. The middle of the lane change may have a secondprogress threshold 214, T1, indicating that approximately 50% of thelane change has been completed. At the second progress threshold 214,T1, the value of the left control operand 106 may be approximately 0.5,50%, or another value indicating that the middle of the lane change hasbeen reached, as indicated by the chart 218. As such, the value for theleft control operand 106 may remain relatively constant throughout thefirst stage 208 (e.g., at 1.0), and may not change (e.g., to 0.5) untilthe second progress threshold 214, T1, is reached.

Similarly, the second stage 210 may be from the middle of the lanechange until the end of the lane change, or a third progress threshold216, T2, indicating the end of the lane change was reached (e.g.,approximately when between 80% to 100% of the lane change has beencompleted). At the third progress threshold 216, T2, the value of theleft control operand 106 may be approximately 0.0, 0%, or another valueindicating that the lane change is ending or ended, as indicated by thechart 218. As such, the second stage 210 of the left lane change mayspan between the second progress threshold 214, T1, and the thirdprogress threshold 216, T2. As such, the value for the left controloperand 106 may remain relatively constant throughout the second stage210 (e.g., at 0.5), and may not change (e.g., to 0.0) until the thirdprogress threshold 216, T2, is reached.

The chart 218 is indicative of the values for the left control operand106 throughout the left lane change. As described above, the value ofthe left control operand 106 may be 1.0 throughout the first stage 208,0.5 throughout the second stage 210, and 0 after (e.g., when the lanechange is complete, or the third progress threshold 216, T2, has beenreached). The chart 218 also includes a lane change progress to leftcontrol operand line 224. The lane change progress to left controloperand line 224 is a representation of the values of the left controloperand 106 if the left control operand 106 were set, updated, and/orchanged based on each lane change progress 122 output by the machinelearning model(s) 118 (e.g., lane change progress 122 of 73% at a firsttime, the value of the left control operand 106 may be set to 0.27, thenlane change progress 122 of 75% at a second time after the first time,the value of the left control operand may be set to 0.25, then lanechange progress 122 of 74% at a third time after the second time, thevalue of the left control operand may be set to 0.26). As a result, andas indicated by the lane change progress to left control operand line224, if the left control operand 106 was set, updated, and/or changedbased on each lane change progress 122 output by the machine learningmodel(s) 118, the result may be noisy.

As such, by implementing stages into lane changes (or other vehiclemaneuvers), these noisy outputs from the machine learning model(s) 118may be avoided and, additionally, the vehicle 140 may be more capable ofdetermining which portion (or stage) of the lane change to driveaccording to. For example, because a lane change includes basically twoturns (e.g., for the left lane change of FIG. 2, a first turn to theleft from the first lane 204A to the second lane 204B, and then a secondturn to the right to re-align/re-orient with the second lane 204B), bycreating two stages, the vehicle 140 is able to determine whether it iscurrently making the first turn from the first lane 204A to the secondlane 204B or the second turn from the middle of the lane change to lineup with the second lane 204B (e.g., to enter into a lane keeping mode).In addition, by creating a second progress threshold 214, T1, at amiddle of the lane change, noisy outputs (e.g., 48% progress through theturn) after the second progress threshold 214, T1, has been met (e.g.,50% or greater progress through the turn has been met) can be discarded,or ignored, and the vehicle 140 may continue on a smoother trajectoryrather than an erratic or sporadic trajectory caused by noisy outputs(e.g., as indicated by the lane change progress to left control operandline 224 of the chart 218).

In some examples, the vehicle 140 may be in a lane keeping mode prior tobeginning the lane change (e.g., the vehicle may be in a mode thatcontrols the vehicle 140 to stay in the first lane 204A, where the lanekeeping mode may include its own machine learning model(s) and controlparameters). During the lane change, the vehicle 140 may be in a lanechange mode and, once the third progress threshold 216, T2, is reached,the vehicle 140 may reenter the lane keeping mode (e.g., the vehicle mayreenter the lane keeping mode, but now for staying in the second lane204B). In some examples, entering the lane changing mode may end priorto fully completing the lane change (e.g., where the third progressthreshold 216, T2, is less than at 100% progress, such as at 75%, 80%,85%, etc.). In such examples, the lane keeping mode may be responsiblefor finishing the lane change by lining the vehicle 140 up with thesecond lane 204B and maintaining a lane keeping trajectory.

Although FIG. 2 is described with respect to a left lane change, this isnot intended to be limiting. For example, the processes described withrespect to FIG. 2 may be implemented for right lane changes, turns, lanesplits, and/or any other vehicle maneuvers.

Now referring to FIG. 3, FIG. 3 is an illustration of an example vehiclemaneuver 200 and example training data for a machine learning model(s)118, in accordance with some embodiments of the present disclosure. FIG.3 may include at least one example of training information 300 that maybe used to train the machine learning model(s) 118 (e.g., theconvolutional network 118A). The training information 300 may includesensor information (e.g., similar to the sensor information 110 of FIGS.1A-1B), such as image data from camera(s), or other sensor data fromLIDAR sensor(s), RADAR sensor(s), and/or the like. In the example ofFIG. 3, the training information 300 may include image data from acamera(s) of the vehicle 140 (or another vehicle used for training themachine learning model(s) 118).

To train the machine learning model(s) 118, images 302 (as representedby image data) generated from a camera(s) of the vehicle 140 during lanechanges may be used. The images 302 may be tagged at various pointsalong the lane changes, such as at the beginning, the middle, and theend of the lane change. Different images 302 may be tagged for left lanechanges and right lane changes.

As illustrated in FIG. 3, the images 302 may be tagged at the beginningof the lane change to correspond with the first progress threshold 212,T0, the middle of the lane change to correspond with the second progressthreshold 214, T1, and the end of the lane change to correspond with thethird progress threshold 216, T2. In addition, the images 302 may betagged with the corresponding left control operand 106 and/or rightcontrol operand 108 value (e.g., image 302A, from the beginning of thelane change, may be tagged with a left control operand 106 value of1.0).

As a result, the machine learning model(s) 118 may learn to more clearlyidentify the stages (e.g., the first stage 208 and the second stage 210)of the lane change and the progress thresholds. By tagging the images302 at the three stages, the machine learning model(s) 118 may alsoextrapolate values of the left control operand 106 and/or the rightcontrol operand 108 over the entire lane change (e.g., for each of theimages between the image 302A and the image 302B, and between the image302B and the image 302C). In other words, by tagging the images 302corresponding to the beginning of the lane change (e.g., image 302A),the middle of the lane change (e.g., image 302B), and the end of thelane change (e.g., image 302C) with corresponding values for the leftcontrol operand 106 and/or the right control operand 108, the machinelearning model(s) 118 may learn values for the left control operand 106and/or the right control operand 108 and tag the remainder of the images302 with their corresponding values.

In some examples, extrapolating of the values of the left controloperand 106 and/or the right control operand 108 for the remainder ofthe images 302 of the training information 300 may be performedseparately from the machine learning model(s) 118. For example, if theimage 302A is tagged with a value of 1.0 for the left control operand106, and the image 302B is tagged with a value of 0.5 for the leftcontrol operand, then each of the images 302 between the image 302A andthe image 302B in the sequence of the lane change may be tagged with avalue for the left control operand 106 using extrapolation (e.g., iffour images 302 are to be tagged, they may be tagged as 0.9, 0.8, 0.7,and 0.6). In some examples, a separate machine learning model(s) may beused to determine the values for the left control operand 106 and/or theright control operand 108 for each of the images 302 between the taggedimages 302 (e.g., the images 302A, 302B, and 302C).

As an example, and without limitation, the image 302A may be chosen asthe beginning of the lane change, and tagged with a value of the leftcontrol operand 106 of 1.0 (e.g., assuming the lane change is the leftlane change of the vehicle maneuver 200), based on the vehicle 140 beingin a first lane 204A (e.g., a right lane) and beginning to make a turntoward the left (e.g., so that the centerline 308 of the image 302A maybe extending through both the first lane 204A and the second lane 204Band through the lane markings dividing the two lanes. In other examples,the beginning of the lane change may be tagged when the centerline 308is aligned only with the first lane 204A, but immediately before thelane change is to begin.

As another example, and without limitation, the image 302B may be chosenas the middle of the lane change, and tagged with a value of 0.5 for theleft control operand 106, based on the vehicle 140 being in between thefirst lane 204A and the second lane 204B such that the centerline 312 ofthe image 302B approximately lines up with the lane markings separatingthe first lane 204A from the second lane 204B, or when the vehicle 140is roughly equidistant to the outer lane markings of the original laneand the destination lane.

As yet another example, and without limitation, the image 302C may bechosen as the end of the lane change, and tagged with a value of 0.0 forthe left control operand 106, based on the vehicle 140 being in thesecond lane 204B that the centerline 316 of the image 302C approximatelylines up with the second lane 204B. In some examples, as describedherein, the lane change may be deemed to be completed prior to thevehicle 140 being fully lined up in the destination lane (e.g., thesecond lane 204B). In such examples, the image 302C may be tagged as theend of the lane change prior to the centerline 316 being lined up withinthe second lane 204B, such as when the centerline is still angled acrossthe second lane 204B and, in the case of a left lane change, the vehicle140 has not fully turned back toward the right to line up with thesecond lane 204B.

Now referring to FIG. 4, FIG. 4 is a flow diagram showing a method 400for performing a vehicle maneuver, in accordance with some embodimentsof the present disclosure. Each block of the method 400, describedherein, comprises a computing process that may be performed using anycombination of hardware, firmware, and/or software. For instance,various functions may be carried out by a processor executinginstructions stored in memory. The methods may also be embodied ascomputer-usable instructions stored on computer storage media. Themethods may be provided by a standalone application, a service or hostedservice (standalone or in combination with another hosted service), or aplug-in to another product, to name a few. In addition, method 400 isdescribed, by way of example, with respect to the vehicle 140 and theprocess 100. However, the method 400 may additionally or alternativelybe executed by any one system, or any combination of systems, including,but not limited to, those described herein.

The method 400, at block B402, includes receiving a trigger signal. Forexample, in response to an input to the vehicle 140 representative of acommand or request to initiate a vehicle maneuver (e.g., a lane change,a turn, etc.) and/or in response to a determination by the vehicle 140(e.g., by a planning layer of the autonomous driving software stack,such as using a GPS application) to execute a vehicle maneuver, thetrigger signal may be generated and received.

The method 400, at block B404, includes receiving sensor data. Forexample, the sensor information 110 may be generated and/or captured byone or more sensors and/or cameras of the vehicle 140 and received. Thesensor information 110 may include sensor data and/or image datarepresentative of a field of view(s) of one or more sensors and/orcameras.

The method 400, at block B406, includes applying a value of a controlinput and the sensor data to a machine learning model(s). For example,the value of the left control operand 106 and/or the right controloperand 108 in addition to the sensor information 110 may be applied tothe machine learning model(s) 118.

The method 400, at block B408, includes computing, by the machinelearning model(s), vehicle control data and progress data. For example,the machine learning model(s) 118 may compute the vehicle controlinformation 124 and/or the lane change progress 122 based at least inpart on the inputs 104 (e.g., the sensor information 110, the value ofthe left control operand 106 and/or the right control operand 108, thestatus information 112, and/or other inputs 104).

The method 400, at block B410, includes transmitting the vehicle controldata to a control component of the vehicle. For example, the vehiclecontrol information 124 may be transmitted to the control component(s)128 of the vehicle 140. As described herein, the vehicle controlinformation 124 may include a recommended vehicle trajectory and/orcontrols for controlling the vehicle 140 according to the recommendedvehicle trajectory (e.g., steering angle, acceleration, etc.).

The method 400, at block B412, includes updating the value of thecontrol input. For example, based at least in part on the outputs 120 ofthe machine learning model(s) 118, the value of the left control operand106 and/or the right control operand 108 may be updated.

The method may continue by repeating block B406 to block B412 until thevehicle maneuver has completed. Once the vehicle maneuver is determinedto have completed, the vehicle 140 may enter another driving mode, suchas a lane keeping mode.

Turns and Lane Splits

As described above, conventional systems rely on HD maps for performingmaneuvers such as turns and/or lane splits at intersections. However, byrelying on an HD map, an autonomous vehicle must first have accuratelocation information (e.g., from a GPS sensor) and an accurate HD map,and perform localization processing to merge the information forreferencing. As a result, when an HD map does not exist and/or when thelocation information is updating slowly or is inaccurate, making turnsor taking lane splits may not be performed effectively or at all (e.g.,the autonomous vehicle may implement a safety protocol to abort themaneuver). Otherwise, the autonomous vehicle may perform the maneuvererratically or dangerously (e.g., because the location information isinaccurate, so the localization of the vehicle with respect to theintersection and the turn and/or lane split may suffer as a result), orrequire hand-off of the controls to a human driver. None of theseoptions are particularly desirable for a passenger of the autonomousvehicle.

The present systems provide for an autonomous vehicle 140 that may makelane changes without relying an HD map and/or location information.Instead, the autonomous vehicle relies on computer vision using sensordata generated from one or more sensors of the vehicle (e.g., cameras,LIDAR sensors, RADAR sensors, etc.). As a result, turns and lane splitsare principally perception based, and thus an over-reliance onpotentially unavailable and/or inaccurate information (e.g., HD maps,location information, etc.) is avoided.

With reference to turns at intersections or lane splits, a triggersignal (e.g., turn right, turn left, take lane split to right, etc.) maybe received in response to an input (e.g., a user input to a turn signalor blinker switch) or otherwise in response to a command from the system(e.g., a navigation command from a GPS system), to indicate that a turnor a lane split is to begin. The trigger signal may cause a controlinput (e.g., a left control operand for turning left, a right controloperand for turning right, a keep right control operand for followingthe lane split right, etc.) to be updated (e.g., for right turn, theright control operand may be set to an initial value, such as 1.0).

Sensor data from one or more sensors of the vehicle (e.g., cameras,LIDAR sensors 864, RADAR sensors 860, etc.) may be received and inputinto a machine learning model(s) (e.g., a convolutional neural networkthat may include individual convolutional streams for each sensor). Insome examples, the machine learning model(s) may output vehicletrajectory points (e.g., (x, y) coordinates) representative of arecommended vehicle trajectory and/or controls for a vehicle controlunit (e.g., control component(s) of the vehicle) for following therecommended vehicle trajectory. The recommended vehicle trajectory maycontinue on the current road or along the current path (e.g., not turn)until a suitable intersection or off-ramp is identified. In other words,a recommended vehicle trajectory for a turn or lane split may includenot only locations for the turn or lane split but also a stretch of road(which can be indefinitely long) prior to and/or after the turn or lanesplit locations.

In some examples, orientation information of the vehicle may be used todetermine that the turn or lane split has been completed. For example,if the turn is a 90-degree turn, then a change in orientation of thevehicle of 90 degrees (as measured by any number or combination ofsuitable sensors such as, and without limitation, IMU sensors) mayindicate to the system that the turn is complete. In such examples, theorientation information (e.g., as status information) may be input intothe machine learning model(s) and the machine learning model(s) mayoutput the status of the turn. In other examples, the orientationinformation may be separately calculated (e.g., not using the machinelearning model(s), or using another machine learning model(s)).

In addition, the machine learning model(s) may output turn or lane splitstatus information (e.g., the amount of the turn or lane split that hasbeen completed) and this output may be fed back to the left controloperand or the right control operand (e.g., dependent on the directionof the turn or lane split). This process may continue until the turn orlane split is completed. Once the turn or lane split is complete, thecontrol input may be reset (e.g., set to 0.0, to indicate that the turnor lane split is complete).

In some examples, map data representing a basic depiction of anintersection (e.g., a screenshot of the intersection from a 2D GPSapplication) may be input into the machine learning model(s) in additionto the sensor data. The machine learning model(s) may use the map datato generate the recommended vehicle trajectory that includes making aturn once an intersection that corresponds to the map data is identifiedfrom the sensor data. As a result, a detailed (e.g., HD) map, as used byconventional systems, is not required to obtain the benefit of map data,thereby reducing the requirements that HD map data needs to be availableand accurate when controlling the vehicle through a turn or lane split.

In other examples, data representative of a low-resolution map may beinput into the machine learning model(s). This map data may represent a2D road layout, a path along the road layout for the vehicle to follow(e.g., a GPS guided path based on a current location of the vehicle andan end location), and/or a location of the vehicle with respect to oneof the road layout or the path (e.g., with the vehicle at the center,oriented vertically). In such examples, the map data may be capturedand/or input to the machine learning model(s) at each iteration. The mapdata may be used for basic localization for the vehicle 140, such as toidentify an estimated relative location of the vehicle 140 with respectto the map.

In addition, in such examples, a control input may not be needed,because a turn or lane split is a part of a continuous path from astarting location to an end location and, therefore, by following thepath, the vehicle may not need to be blindly (e.g., without anysupporting data) identifying an intersection since the intersection isidentified in the map data. However, the output of the machine learningmodel(s) may still include the vehicle trajectory points or controls forfollowing the vehicle trajectory.

In any example, the sensor data, the basic map data, and/or the moredetailed map data may be applied to a convolutional neural network.Sensor data from each sensor may be applied to a respectiveconvolutional stream, and the map data may be applied to yet anotherconvolutional stream. The outputs of the convolutional streams may becombined at a layer of the convolutional neural network, such as at thefully connected layer.

Now referring to FIG. 5A, FIG. 5A is a data flow diagram illustrating anexample process 500 for performing a vehicle maneuver, in accordancewith some embodiments of the present disclosure. The vehicle maneuverdescribed with respect to FIG. 5A is a turn or lane split, however, asdescribed herein, this is for example purposes only and is not intendedto be limiting.

The vehicle maneuver may begin when a trigger signal from, innon-limiting examples, a signal generator 502 is received. Any number ofinputs 504, including but not limited to those illustrated in FIG. 5A,may be input into a machine learning model(s) 516. The machine learningmodel(s) 516 may generate or compute any number of outputs 518,including but not limited to those illustrated in FIG. 5A. At least oneof the outputs 518 may be fed back into the inputs 504, as indicated bythe feedback loop 522 (e.g., the outputs 518 may be used to update oneor more of the inputs 504 for a next iteration of the machine learningmodel(s) 516. At least one of the outputs 518 may be transmitted or sentto a control component(s) 128 of the autonomous vehicle 140. The controlcomponent(s) 128 may then use the output(s) 518 (or informationgenerated from the output(s) 518) to control the vehicle 140 accordingto the output(s) 518.

The trigger signal may include data representative of the type ofvehicle maneuver to be completed by the vehicle 140. For example, thetrigger signal may include data representing a right turn, a left turn,a left lane split, a right lane split, and/or another vehicle maneuver,such as a lane change, as described in more detail above with respect toFIGS. 1A-1B. In some examples, the trigger signal may be generated usingpulse width modulation (PWM), or another signal generation method, andmay be analyzed by a component(s) of the vehicle 140 to determine thetype of vehicle maneuver represented by the trigger signal.

The trigger signal may be received or generated in response to an inputreceived by the vehicle 140 and/or in response to a decision made by thevehicle 140. For example, an operator of the vehicle 140 may provide aninput via a turn signal, a keyboard, a human machine interface (HMI)display, voice input, and/or another input type. The input may berepresentative of a specific vehicle maneuver and, as a result, thetrigger signal representing the specific vehicle maneuver may begenerated. As another example, the vehicle 140 (e.g., one or morecomponents or features of a planning layer of an autonomous vehiclesoftware stack) may determine that a specific vehicle maneuver isrequired or desired and, as a result, may generate the trigger signalrepresenting the specific vehicle maneuver. In such an example, thevehicle 140 may be using an HD map, navigational guidance, and/oranother information source to determine that the specific vehiclemaneuver is required or desired (e.g., to increase safety). In someexamples, the trigger signal may be a guidance signal from a GNSSapplication (e.g., a GPS application).

The inputs 504 may include a left control operand 506, a right controloperand 508, sensor information 510, status information 512, mapinformation 514, and/or other inputs. Values of the left control operand506 and the right control operand 508 may be determined based on thetrigger signal and/or feedback from the outputs 518 of the machinelearning model(s) 516. For example, when the trigger signal isrepresentative of a left turn, the value of the left control operand 506may be set, updated, and/or changed based on the trigger signal. Asanother example, when the trigger signal is representative of a rightturn, the value of the right control operand 508 may be set, updated,and/or changed based on the trigger signal. In any example, the machinelearning model(s) 516 may use the left control operand 506 and/or theright control operand 508 in computing the vehicle control information520 and/or other outputs 518.

In some examples, in response to receiving the trigger signal, the valueof the left control operand 506 and/or the right control operand 508 maybe set, updated, and/or changed to 1, 100%, and/or another valueindicating a start of the turn or lane split. At the end of the turn orlane split, the value of the left control operand 506 and/or the rightcontrol operand 508 may be set, updated, and/or changed to 0, 0%, and/oranother value indicating that the turn or lane split is complete. Insome examples, the value of the left control operand 506 and/or theright control operand 508 may be updated throughout the turn or lanesplit, similar to the updating of the left control operand 106 and/orthe right control operand 108 for a lane change, as described herein.However, because orientation information (e.g., the status information512) and/or map information 514 may be used, the left control operand506 and/or the right control operand 508 may only need to be updated toindicate that the turn or lane split should start and that the turn orlane split should end (e.g., so that the vehicle 140 can reenter anothermode, such as a lane keeping mode).

Another input 504 that may be applied to the machine learning model(s)516 is sensor information 510. The sensor information 510 may includesensor data from any of the sensors of the vehicle 140 (and/or othervehicles, in some examples). For example, with reference to FIGS. 8A-8C,the sensor information 110 may include the sensor data generated by, forexample and without limitation, global navigation satellite systems(GNSS) sensor(s) 858 (e.g., Global Positioning System sensor(s)), RADARsensor(s) 860, ultrasonic sensor(s) 862, LIDAR sensor(s) 864, inertialmeasurement unit (IMU) sensor(s) 866 (e.g., accelerometer(s),gyroscope(s), magnetic compass(es), magnetometer(s), etc.),microphone(s) 896, stereo camera(s) 868, wide-view camera(s) 870 (e.g.,fisheye cameras), infrared camera(s) 872, surround camera(s) 874 (e.g.,360 degree cameras), long-range and/or mid-range camera(s) 898, speedsensor(s) 844 (e.g., for measuring the speed of the vehicle 140),vibration sensor(s) 842, steering sensor(s) 840, brake sensor(s) (e.g.,as part of the brake sensor system 846), and/or other sensor types.

In some examples, the sensor information 510 may include the sensor datagenerated by forward-facing and/or side-facing cameras, such as awide-view camera(s) 870, a surround camera(s) 874, a stereo camera(s)868, and/or a long-range or mid-range camera(s) 898. This sensor datamay be useful for computer vision and/or perception when navigating aturn because a forward-facing and/or side-facing camera may include afield of view (e.g., the field of view of the forward-facing stereocamera 868, the forward-facing long range camera(s) 898, and/or thewide-view camera 870 of FIG. 8B) that includes both a current road oftravel of the vehicle 140 and at least a portion of the intersection orroad that the vehicle 140 is to turn into to complete the vehiclemaneuver. In some examples, more than one camera or other sensor may beused to incorporate multiple fields of view (e.g., the field of view ofthe long-range cameras 898, the forward-facing stereo camera 868, and/orthe forward facing wide-view camera 870 of FIG. 8B).

For example, a first camera may be used that has a field of viewextending along the centerline of the vehicle 140 (e.g., theforward-facing stereo camera 868), a second camera with a field of viewextending along a line that is approximately 45 degrees offset to theleft of the field of view of first camera (e.g., the left-forward-facinglong-range camera 898), and a third camera with a field of viewextending along a line that is approximately 45 degrees offset to theright of the field of view of the first camera (e.g., theright-forward-facing long-range camera 898). The first camera and thethird camera may be mounted on or integrated in the wing mirrors, or maybe mounted on or integrated in the body of the vehicle 140 along thewindshield, for example. The second camera may be mounted or integratedabove the windshield on the body of the vehicle 140, may be mounted onthe rear-side of the rear-view mirror internal to the vehicle 140,and/or may be mounted on or integrated in the bumper or another regionof the front body of the vehicle 140.

As another example, a single camera that includes a larger field of view(e.g., a field of view that includes at least portions of the fields ofview of the first camera, the second camera, and the third camera) maybe used, such as the wide view camera 870. In any example, a camera(s)that includes a field(s) of view that encompasses at least 120 degreesto 180 degrees in front of the vehicle 140 may be useful for navigatingturns and/or lane splits using the machine learning model(s) 516.

In any example, the sensor information 510 may include image datarepresenting image(s), image data representing video (e.g., snapshots ofvideo), and/or sensor data representing fields of view of sensors (e.g.,LIDAR sensor(s) 864, RADAR sensor(s) 860, etc.). The sensor information510 may be input into the machine learning model(s) 516 and used by themachine learning model(s) 516 to compute the outputs 518, such as thevehicle control information 520.

The inputs 504 may include status information 512. The statusinformation 512 may include data representative of the status of thevehicle 140, such as speed, velocity, acceleration, deceleration,orientation, and/or other status information. This data may be capturedby and/or received from one or more of the sensors of the vehicle 140,such as one or more of the IMU sensor(s) 866, speed sensor(s) 844,steering sensor(s) 840, vibration sensor(s) 842, and/or one or moresensors of the brake sensor system 846, propulsion system 850, and/orsteering system 854.

The status information 512 may be used by the machine learning model(s)516 to compute the outputs 518, such as the vehicle control information520. For example, when the vehicle control information 520 includes arecommended vehicle trajectory, or control data for controlling thevehicle 140 according to a recommended vehicle trajectory, the statusinformation 512 (e.g., speed, orientation, etc.) may be valuable to themachine learning model(s) 516 in performing such computations. In otherwords, a vehicle trajectory that is representative of recommended future(x, y) coordinates of the vehicle 140 may benefit from knowledge of thecurrent speed, orientation, and/or other status information 512 of thevehicle 140. Further, in order to determine that a turn or lane splithas been completed, the status information 512 (e.g., orientationinformation) may be useful (e.g., if a turn is 90 degrees, thendetermining that the orientation of the vehicle 140 has changed byapproximately 90 degrees may indicate that the turn is complete).

Further, in some examples, the inputs 504 may include map information514. The map information 514 may be used by the machine learningmodel(s) 516 to generate the outputs 518, such as the vehicle controlinformation 520. For example, the map information 514 may includelow-resolution map data (e.g., screenshots of a 2D map application withor without guidance). In other words, in contrast with map datarepresenting an HD map (e.g., the HD map 822 and/or the HD mapsdescribed herein and used by conventional systems), the map information514 may require significantly less storage space, and used only as anadditional data point by the machine learning model(s) 516 whencomputing the outputs 518. The map information 514 may include ascreenshot or an image (or data representative thereof) that depicts anintersection (or at least a lane split or turn), such as illustrated bymap information 514A of FIG. 5B. In such an example, the map information514 may include a single image or screenshot, or data representativethereof, of an intersection, and may be provided to the machine learningmodel(s) 516 once a navigational waypoint has been reached (e.g., oncean indication has been received from a GPS application that the turn orlane split is within a certain distance).

In other examples, the map information 514 may include other map data,such as a path along the road(s) and/or intersection(s) that indicatesguidance (e.g. GPS guidance) for the vehicle through the road(s) and/orintersection(s), as illustrated by map information 514B of FIG. 5C. Insuch examples, the map information 514 may include multiple images orscreenshots, or data representative thereof, of the road, the vehicle140, and/or a path. For example, at each iteration of the machinelearning model(s) 516, new map information 514 may be generated to beused by the machine learning model(s) 516 in addition to at least thesensor information 512 to determine the vehicle control information 520.

In any example, the map information 514 may be generated by the vehicle140, such as by a GNSS application of the vehicle (e.g., GPS) or anotherapplication of the vehicle (e.g., as part of the planning layer of anautonomous driving software stack), may be generated by and receivedfrom a server (e.g., the server(s) 878 of FIG. 8D), may be generated byand received from a client device (e.g., a mobile phone, a computer,etc.), and/or may be received and/or generated by another method.

In examples where the map information 514 is one of the inputs 504, andthe map information 514 includes the path (e.g., a GPS guidance path)for navigating through the vehicle maneuver, the left control operand506 and/or the right control operand 508 may be used differently or notat all. For example, the left control operand 506 and/or the rightcontrol operand 508 may only include a start value (e.g., 1, 100%, etc.)for the start of the turn or lane split and an end value (e.g., 0, 0%,etc.) for the end of the turn or lane split. In other examples, the leftcontrol operand 506 and/or the right control operand 508 may not beneeded. In either example, the left control operand 506 and/or the rightcontrol operand 508 may be different because the vehicle's location withrespect to the path may indicate the progress of the vehicle 140 throughthe turn or lane split, so the machine learning model(s) 516 may notrequire the left control operand 506 and/or the right control operand508 when determining whether the turn or lane split is complete (e.g.,the vehicle 140 may rely on the path to make this computation instead).

The machine learning model(s) 516 may use the inputs 504 to compute theoutputs 518. Although examples are described herein with respect tousing neural networks, and specifically convolutional neural networks,as the machine learning model(s) 516 (e.g., with respect to FIGS.5B-5C), this is not intended to be limiting. For example, and withoutlimitation, the machine learning model(s) 516 may include any type ofmachine learning model(s), such as machine learning models using linearregression, logistic regression, decision trees, support vector machines(SVM), Naïve Bayes, k-nearest neighbor (Knn), K means clustering, randomforest, dimensionality reduction algorithms, gradient boostingalgorithms, neural networks (e.g., auto-encoders, convolutional,recurrent, perceptrons, long/short terms memory, Hopfield, Boltzmann,deep belief, deconvolutional, generative adversarial, liquid statemachine, etc.), and/or other types of machine learning models.

The outputs 518 may include vehicle control information 520, and/orother output types, such as progress through the turn or lane split (orat least an end of a turn or lane split, to update the left controloperand 506 and/or the right control operand 508). The vehicle controlinformation 520 may include a recommended vehicle trajectory and/orvehicle control data for controlling the vehicle according to therecommended vehicle trajectory. The outputs 518 may include one or morepoints (e.g., (x, y) coordinates) along the recommended vehicletrajectory and/or may include the recommended vehicle trajectory (e.g.,a trajectory extrapolated over each of the points). In some examples,the recommended vehicle trajectory may be represented as a radius of thevehicle maneuver, while in other examples, the recommended vehicletrajectory may be represented as an inverse radius of the vehiclemaneuver. The inverse radius may be used in some examples to prevent therecommended vehicle trajectory (or a point thereof) from being computedas an infinite value (e.g., reaching singularity).

In examples where the vehicle control information 520 includes therecommended vehicle trajectory, the recommended vehicle trajectory (ordata representative thereof) may be sent or transmitted to the controlcomponent(s) 128 of the vehicle 140 (e.g., to a control layer of theautonomous driving software), and the control component(s) 128 maydetermine the controls required to control the vehicle 140 according tothe recommended vehicle trajectory. For example, the controlcomponent(s) 128 may send one or more controls to one or more actuators(e.g., actuators controlled by an actuation layer of the autonomousdriving software stack). The actuators may include one or morecomponents or features of the brake sensor system 846, the propulsionsystem 850, the steering system 854, and/or other systems. The vehicle140 may then be controlled according to the recommended vehicletrajectory of the vehicle control information 520 output by the machinelearning model(s) 516. By only outputting the recommended vehicletrajectory and not the controls themselves, the process 500 may be morelikely to be used by different vehicle manufacturers because thedifferent vehicle manufacturers may determine their own controls andactuations for controlling the vehicle 140 according to the recommendedvehicle trajectory.

In other examples, the vehicle control information 520 may include thecontrols for controlling the vehicle 140 according to the recommendedvehicle trajectory. In such examples, the machine learning model(s) 516may be implemented at the control layer of the autonomous drivingsoftware stack, and the vehicle control information 520 may be used tocontrol the vehicle 140 (e.g., cause actuation of one or more actuatorsof the vehicle by the actuation layer of the autonomous driving softwarestack).

Now referring to FIG. 5B, FIG. 5B is an illustration of an examplemachine learning model(s) 516A, in accordance with some embodiments ofthe present disclosure. The machine learning model(s) 516A of FIG. 5Bmay be one example of a machine learning model(s) 516 that may be usedin the process 500. However, the machine learning model(s) 516A of FIG.5B is not intended to be limiting, and the machine learning model(s) 516may include additional and/or different machine learning models than themachine learning model(s) 516A of FIG. 5B. The machine learning model(s)516A may include a convolutional neural network and thus mayalternatively be referred to herein as convolutional neural network 516Aor convolutional network 516A.

The convolutional network 516A may include inputs 504 that include theleft control operand 506, the right control operand 508, the sensorinformation 510, the map information 514A, and/or other input types. Theconvolutional network 516A may include sensor information 510A-510Cwhich may include image data generated by one or more cameras (e.g., oneor more of the cameras described herein with respect to FIGS. 8A-8C).For example, the sensor information 510A may include image datagenerated by a first camera (e.g., a left-forward-facing camera), thesensor information 510B may include image data generated by a secondcamera (e.g., a center-forward-facing camera), and the sensorinformation 510C may include image data generated by a third camera(e.g., a right-forward-racing camera), such as the first, second, andthird cameras described above. More specifically, the sensor information510A-510C may include individual images generated by the camera(s),where image data representative of one or more of the individual imagesis input into the convolutional network 516A at each iteration of theconvolutional network 516A.

The sensor information 510 may be input into convolutional stream(s) 526of the convolutional network 516A. For example, the sensor information510 from each sensor (e.g., camera, LIDAR sensor, RADAR sensor, etc.)may be input into its own convolutional stream 526 (e.g., the sensorinformation 510A may be input into the convolutional stream 526A, thesensor information 510B may be input into the convolutional stream 526B,and so on). The convolutional stream(s) 526 and/or the convolutionalnetwork 516A may include any number of layers 528, 530, and/or 532,parameters, and/or hyper-parameters. The layers 528, 530, and/or 532,parameters, and/or hyper-parameters may be similar to those describedabove with respect to the convolutional stream 130 and/or theconvolutional network 118A of FIG. 1B (e.g., convolutional layers,pooling layers, ReLU layers, etc.).

To train the machine learning model(s) 516, the sensor information 510may be tagged. For example, the sensor information 510 indicative of abeginning of a turn or lane split may be tagged with a value for theleft control operand 506 and/or the right control operand 508 indicatingthat the turn or lane split has begun (e.g., 1.0, 100%, etc.). Thesensor information 510 indicative of an end of the turn or lane splitmay be tagged with a value for the left control operand 506 and/or theright control operand 508 indicating that the turn or lane split hasended (e.g., 0.0, 0%, etc.). Similar to the training information 300described above with respect to FIG. 3, the remainder of the sensorinformation between the start tag and the end tag may be tagged based onextrapolation (which may be computed using rules, logic, the machinelearning model(s) 516, and/or another machine learning model(s)).

In some examples, the tagging of the start of the turn or lane split maybe prior to the actual turn or lane split (e.g., an instance of thesensor information 510, such as an image, that is 100 feet, 200 feet, 10frames, 20 frames, etc. before the actual turn or lane split starts maybe tagged as the start of the turn or lane split). Similarly, thetagging of the end of the turn or lane split may be after the actualturn or lane split has finished (e.g., an instance of the sensorinformation, such as an image, that is 100 feet, 150 feet, 25 frames, 30frames, etc. after the actual turn or lane split ends may be tagged asthe end of the turn or lane split). In such examples, the turn or lanesplit may include a first portion prior to the turn or lane split, asecond portion during the turn or lane split, and/or a third portionafter the turn or lane split. By doing this, the machine learningmodel(s) 516 may learn to identify the turn or lane split (e.g., theintersection) as part of the turn or lane split.

In some examples, the map information 514A may be input intoconvolutional stream(s) 527 of the convolutional network 516A. The mapinformation 514A and/or the convolutional stream(s) 527 may be omittedfrom certain embodiments (as indicated by the dashed lines). In exampleswhere they are not omitted, the map information 514A may be a 2D,low-resolution representation of an intersection 536 where the turn orlane change is performed. The convolutional stream(s) 527 may alsoinclude any number of layers 533, parameters, and/or hyper-parameters.The layers 533, parameters, and/or hyper-parameters may be similar tothose described above with respect to the convolutional stream 130and/or the convolutional network 118A of FIG. 1B (e.g., convolutionallayers, pooling layers, ReLU layers, etc.).

The output of the convolutional stream(s) 526 (and the convolutionalstream(s) 527, in some examples) may be input to a fully connectedlayer(s) 534 of the convolutional network 516A. In addition to theoutput of the convolutional stream(s) 526 (and 527), the left controloperand 506, the right control operand 508, and/or one or more otherinputs 504 may be input to the fully connected layer(s) 534.

The outputs 518 of the convolutional network 516A may include thevehicle control information 520, and/or other output types, such asprogress of the turn or lane split that may be fed back to the leftcontrol operand 506 and/or the right control operand 508, as describedherein. The vehicle control information 520, as described herein, mayinclude a recommended vehicle trajectory 542 and/or controls forfollowing the recommended vehicle trajectory 542 (e.g., for controllingthe vehicle 140 according to the recommended vehicle trajectory 542,such as steering angle, acceleration, deceleration, etc.). The vehiclecontrol information 520 may include, in some examples, a trajectorypoint(s) 540 (e.g., as represented by (x, y) coordinates) along therecommended vehicle trajectory 542. In some examples, only a singletrajectory point 540 (e.g., the next trajectory point for the vehicle140 to be controlled to) may be output by the machine learning model(s)516A. In other examples, more than one trajectory point 540 may beoutput. As another example, an entire trajectory may be output, whichmay be extrapolated from two or more trajectory points 540. In anyexample, the recommended vehicle trajectory 542 may be output as aradius of the recommended vehicle trajectory 542, or may be output as aninverse radius of the recommended vehicle trajectory 542, as describedherein. The recommended vehicle trajectory 542 and/or the trajectorypoint(s) 540 thereon, may be used by the control component(s) 128 tocontrol the vehicle 140 through the intersection 536 (e.g., to make aright turn as indicated in FIG. 5B).

With reference to the turn progress, the turn progress may be output asdata representative of the turn progress, and the data may be analyzedto determine the turn progress (and/or the lane split progress). Inother words, the output of the machine learning model(s) 516 may not bethe value of the turn progress, but may instead be a value thatcorrelates to the turn progress. For example, a lookup table may be usedto determine the turn progress based on the output of the machinelearning model(s) 516.

In some examples, the status information 512 (e.g., the orientation ofthe vehicle 140) may be used to determine the turn progress. In suchexamples, the expected change in orientation of the vehicle 140 throughthe turn or lane split may be known (e.g., from GPS data or from anothersource, such as a source from the planning layer of the autonomousdriving software stack). As such, the expected change in orientation ascompared to the actual change in orientation may be indicative of theturn progress. The control logic 544 may determine when the turn isfinished, and may update the value of the left control operand 506and/or the right control operand 508 in response, as indicated byfeedback loop 522. In some examples, the left control operand 506 mayinclude positive values (e.g., 0-1, 0%-100%, etc.), and the rightcontrol operand 508 may include negative values (e.g., −1-0, −100%-0%,etc.), or vice versa. In such examples, the left control operand 506 andthe right control operand 508 may be a single control (e.g., a universalcontrol), and may range from −1 to 1, or −100% to 100%, and/or the like.

In other examples, the machine learning model(s) 516 may not be used todetermine the turn progress. For example, similar to described above,the expected change in orientation may be known, and sensor data fromone or more IMU sensor(s) 866, for example, may be used to determinewhen the turn or lane split is finished. This computation may beperformed based on rules, logic, and/or may be performed by anothermachine learning model(s) trained to determine orientation changes. Insuch examples, the left control operand 506 and/or the right controloperand 508 may be updated based on this separate computation and maynot be a feedback loop from the machine learning model(s) 516A.

Now referring to FIG. 5C, FIG. 5C is an illustration of another examplemachine learning model(s) 516B, in accordance with some embodiments ofthe present disclosure. The machine learning model(s) 516B of FIG. 5Cmay be one example of a machine learning model(s) 516 that may be usedin the process 500. However, the machine learning model(s) 516B of FIG.5C is not intended to be limiting, and the machine learning model(s) 516may include additional and/or different machine learning models than themachine learning model(s) 516B of FIG. 5C. The machine learning model(s)516B may include a convolutional neural network and thus mayalternatively be referred to herein as convolutional neural network 516Bor convolutional network 516B.

The convolutional network 516B may be similar to that of theconvolutional network 516A, described above (e.g., may include at leastsome similar inputs 504 and/or outputs 518). However, instead of the mapdata 514A, the convolutional network 516B may include the mapinformation 514B which, as described herein, may include the path 546for the vehicle 140 when performing the vehicle maneuver and/or may beapplied to the convolutional network 516B more frequently (e.g., at eachiteration) than the map data 514A of the convolutional network 516A. Assuch, the convolutional network 516B may not need the left controloperand 506, the right control operand 508, and/or the feedback loop 522for updating the left control operand 506 and/or the right controloperand 508, because the path 546 may provide an indication that theturn or lane shift is complete.

For example, the path 546 may include a continuous path from a startingpoint to a destination, and along the way there may be turns, lanesplits, lane changes, etc. In such an example, the convolutional network516B may use the map information 514B, as applied to the convolutionalstream(s) 527, to determine when the turn, lane split, lane change,and/or other vehicle maneuver begins and ends. However, the path 546 isnot intended to include exact coordinates for the vehicle 140 to follow,and thus the vehicle 140 may still rely on the convolutional network516B to provide the recommended vehicle trajectory 542 and/or thecontrol for controlling the vehicle 140 according to the recommendedvehicle trajectory 542. In other words, the path 546 serves as alow-definition, high-level guide for the vehicle 140, and the outputs518 of the convolutional network 516B are used for determining theactual trajectory of the vehicle when navigating turns, lane splits,and/or lane changes.

Now referring to FIG. 6, FIG. 6 is a flow diagram showing a method 600for performing another vehicle maneuver, in accordance with someembodiments of the present disclosure. Each block of the method 600,described herein, comprises a computing process that may be performedusing any combination of hardware, firmware, and/or software. Forinstance, various functions may be carried out by a processor executinginstructions stored in memory. The methods may also be embodied ascomputer-usable instructions stored on computer storage media. Themethods may be provided by a standalone application, a service or hostedservice (standalone or in combination with another hosted service), or aplug-in to another product, to name a few. In addition, method 600 isdescribed, by way of example, with respect to the vehicle 140, theprocess 500, and the machine learning model(s) 516A. However, the method600 may additionally or alternatively be executed by any one system, orany combination of systems, including, but not limited to, thosedescribed herein.

The method 600, at block B602, includes receiving a trigger signal. Forexample, in response to an input to the vehicle 140 representative of acommand or request to initiate a vehicle maneuver (e.g., a turn, a lanesplit, etc.) and/or in response to a determination by the vehicle 140(e.g., by a planning layer of the autonomous driving software stack,such as using a GPS application) to execute a vehicle maneuver, thetrigger signal may be generated and/or received.

The method 600, at block B604, includes setting a control input to aninitial value. For example, in response to or based on receiving thetrigger signal, the left control operand 506 and/or the right controloperand 508 may be set to an initial value (e.g., 1, 100%, or anothervalue indicating that the vehicle maneuver is to be initiated or tobegin).

The method 600, at block B606, includes receiving sensor data. Forexample, the sensor information 510 may be generated and/or captured byone or more sensors and/or cameras of the vehicle 140 and received. Thesensor information 510 may include sensor data and/or image datarepresentative of a field of view(s) of one or more sensors and/orcameras.

In some examples, the method 600 may further include receiving mapinformation 514, such as the map information 514A.

The method 600, at block B608, includes applying the sensor data to amachine learning model(s). For example, the sensor information 510 maybe applied to the machine learning model(s) 516 (e.g., the convolutionalneural network 516A). In addition to the sensor data, the value of theleft control operand 106 and/or the right control operand 108 may beapplied to the machine learning model(s) 516. In some examples, asdescribed herein, the map information 514 (e.g., the map information514A) may be applied to the machine learning model(s) 516.

The method 600, at block B610, includes determining vehicle controldata. For example, the machine learning model(s) 516 may compute and/ordetermine the vehicle control information 520 based on the inputs 504(e.g., the sensor information 510, the map information 514, the statusinformation 512, the left control operand 506, the right control operand508, and/or other inputs 504).

The method 600, at block B612, includes transmitting the vehicle controldata to a control component of a vehicle. For example, the vehiclecontrol information 520 may be transmitted to the control component(s)128 of the vehicle 140. The method 600, at block B612, may be similar tothat of block B412 of the method 400, described above.

The method 600 may include blocks B606 to B612 repeating until thevehicle maneuver is completed. Once the vehicle maneuver is completed,the method 600, at block B614, includes updating the initial value ofthe control input to an end value. For example, the value of the leftcontrol operand 506 and/or the right control operand 508 may be set,updated, and/or changed to indicate to the machine learning model(s) 516that the vehicle maneuver is complete (as indicated by the feedback loop522 of FIG. 5B). The vehicle maneuver may be determined to be completebased on orientation information of the vehicle 140, as determined bythe machine learning model(s) 516, rules, logic, and/or another machinelearning model(s). Once the vehicle maneuver is complete, the vehicle140 may enter another mode, such as a lane keeping mode, as describedherein.

FIG. 7 is a flow diagram showing a method 700 for performing anothervehicle maneuver, in accordance with some embodiments of the presentdisclosure. Each block of the method 700, described herein, comprises acomputing process that may be performed using any combination ofhardware, firmware, and/or software. For instance, various functions maybe carried out by a processor executing instructions stored in memory.The methods may also be embodied as computer-usable instructions storedon computer storage media. The methods may be provided by a standaloneapplication, a service or hosted service (standalone or in combinationwith another hosted service), or a plug-in to another product, to name afew. In addition, method 700 is described, by way of example, withrespect to the vehicle 140, the process 500, and the machine learningmodel(s) 516B. However, the method 700 may additionally or alternativelybe executed by any one system, or any combination of systems, including,but not limited to, those described herein.

The method 700, at block B702, includes receiving map data. For example,the map information 514B may be received. The map information 514B mayinclude the path 546, and the path may include a turn, a lane split, alane change, and/or another vehicle maneuver. The map data 514B may begenerated by and received from a GPS application or another module of aplanning layer of the autonomous driving software stack, for example.

The method 700, at block B704, includes receiving sensor data. Forexample, the sensor information 510 may be generated and/or captured byone or more sensors and/or cameras of the vehicle 140 and received. Thesensor information 510 may include sensor data and/or image datarepresentative of a field of view(s) of one or more sensors and/orcameras.

The method 700, at block B706, includes applying the sensor data and themap data to a machine learning model(s). For example, the sensorinformation 510 and the map information 514 (e.g., the map data 514B)may be applied to the machine learning model(s) 516 (e.g., theconvolutional neural network 516B).

The method 700, at block B708, includes determining vehicle controldata. For example, the machine learning model(s) 516 may compute and/ordetermine the vehicle control information 520 based on the inputs 504(e.g., the sensor information 510, the map information 514, the statusinformation 512, and/or other inputs 504).

The method 700, at block B710, includes transmitting the vehicle controldata to a control component of a vehicle. For example, the vehiclecontrol information 520 may be transmitted to the control component(s)128 of the vehicle 140. The method 600, at block B612, may be similar tothat of block B412 of the method 400, described above.

The method 700 may include blocks B702 to B710 repeating until thevehicle maneuver is completed. Once the vehicle maneuver is complete,the vehicle 140 may enter another mode, such as a lane keeping mode, asdescribed herein.

Example Autonomous Vehicle

FIG. 8A is an illustration of an example autonomous vehicle 140, inaccordance with some embodiments of the present disclosure. The vehicle140 may include a passenger vehicle, such as a car, a truck, a bus,and/or another type of vehicle that may or may not accommodate one ormore passengers. Autonomous vehicles may be generally described in termsof automation levels, such as those defined by the National HighwayTraffic Safety Administration (NHTSA), a division of the US Departmentof Transportation, and the Society of Automotive Engineers (SAE)“Taxonomy and Definitions for Terms Related to Driving AutomationSystems for On-Road Motor Vehicles” (Standard No. J3016-201806,published on Jun. 15, 2018, Standard No. J3016-201609, published on Sep.30, 2016, and previous and future versions of this standard). Thevehicle 140 may be capable of functionality in accordance with one ormore of Level 3-Level 5 of the autonomous driving levels. For example,the vehicle 140 may be capable of conditional automation (Level 3), highautomation (Level 4), and/or full automation (Level 5), depending on theembodiment.

The vehicle 140 may include components such as a chassis, a vehiclebody, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and othercomponents. The vehicle 140 may include a propulsion system 850, such asan internal combustion engine, hybrid electric power plant, anall-electric engine, and/or another propulsion system type. Thepropulsion system 850 may be connected to a drive train of the vehicle140, which may include a transmission, to enable the propulsion of thevehicle 140. The propulsion system 850 may be controlled in response toreceiving signals from a throttle/accelerator 852.

A steering system 854, which may include a steering wheel, may be usedto steer the vehicle 140 (e.g., along a desired path, route, ortrajectory) when the propulsion system 850 is operating (e.g., when thevehicle is in motion). The steering system 854 may receive signals froma steering actuator 856. The steering wheel may be optional for fullautomation (Level 5) functionality.

The brake sensor system 846 may be used to operate the vehicle brakes inresponse to receiving signals from the brake actuators 848 and/or brakesensors.

Controller(s) 836, which may include one or more system(s) on chips(SoCs) 804 (FIG. 8C) and/or GPU(s), may provide signals (e.g.,representative of commands) to one or more components and/or systems ofthe vehicle 140. For example, the controller(s) may send signals tooperate the vehicle brakes via one or more brake actuators 848, tooperate the steering system 854 via one or more steering actuators 856,and/or to operate the propulsion system 850 via one or morethrottle/accelerators 852. The controller(s) 836 may include one or moreonboard (e.g., integrated) computing devices (e.g., supercomputers) thatprocess sensor signals, and output operation commands (e.g., signalsrepresenting commands) to enable autonomous driving and/or to assist ahuman driver in driving the vehicle 140. The controller(s) 836 mayinclude a first controller 836 for autonomous driving functions, asecond controller 836 for functional safety functions, a thirdcontroller 836 for artificial intelligence functionality (e.g., computervision), a fourth controller 836 for infotainment functionality, a fifthcontroller 836 for redundancy in emergency conditions, and/or othercontrollers. In some examples, a single controller 836 may handle two ormore of the above functionalities, two or more controllers 836 mayhandle a single functionality, and/or any combination thereof.

The controller(s) 836 may provide the signals for controlling one ormore components and/or systems of the vehicle 140 in response to sensordata received from one or more sensors (e.g., sensor inputs). The sensordata may be received from, for example and without limitation, globalnavigation satellite systems sensor(s) 858 (e.g., Global PositioningSystem sensor(s)), RADAR sensor(s) 860, ultrasonic sensor(s) 862, LIDARsensor(s) 864, inertial measurement unit (IMU) sensor(s) 866 (e.g.,accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s),etc.), microphone(s) 896, stereo camera(s) 868, wide-view camera(s) 870(e.g., fisheye cameras), infrared camera(s) 872, surround camera(s) 874(e.g., 360 degree cameras), long-range and/or mid-range camera(s) 898,speed sensor(s) 844 (e.g., for measuring the speed of the vehicle 140),vibration sensor(s) 842, steering sensor(s) 840, brake sensor(s) (e.g.,as part of the brake sensor system 846), and/or other sensor types.

One or more of the controller(s) 836 may receive inputs (e.g.,represented by input data) from an instrument cluster 832 of the vehicle140 and provide outputs (e.g., represented by output data, display data,etc.) via a human-machine interface (HMI) display 834, an audibleannunciator, a loudspeaker, and/or via other components of the vehicle140. The outputs may include information such as vehicle velocity,speed, time, map data (e.g., the HD map 822 of FIG. 8C), location data(e.g., the vehicle's 140 location, such as on a map), direction,location of other vehicles (e.g., an occupancy grid), information aboutobjects and status of objects as perceived by the controller(s) 836,etc. For example, the HMI display 834 may display information about thepresence of one or more objects (e.g., a street sign, caution sign,traffic light changing, etc.), and/or information about drivingmaneuvers the vehicle has made, is making, or will make (e.g., changinglanes now, taking exit 34B in two miles, etc.).

The vehicle 140 further includes a network interface 824 which may useone or more wireless antenna(s) 826 and/or modem(s) to communicate overone or more networks. For example, the network interface 824 may becapable of communication over Long-Term Evolution (LTE), WidebandCode-Division Multiple Access (WCDMA), Universal MobileTelecommunications Service (UMTS), Global System for Mobilecommunications (GSM), CDMA2000, etc. The wireless antenna(s) 826 mayalso enable communication between objects in the environment (e.g.,vehicles, mobile devices, etc.), using local area network(s), such asBluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc., and/or LowPower Wide-Area Network(s) (LPWANs), such as Long Range Wide-AreaNetwork (LoRaWAN), SigFox, etc.

FIG. 8B is an example of camera locations and fields of view for theexample autonomous vehicle 140 of FIG. 8A, in accordance with someembodiments of the present disclosure. The cameras and respective fieldsof view are one example embodiment and are not intended to be limiting.For example, additional and/or alternative cameras may be included, fewcameras may be used, and/or the cameras may be located at differentlocations on the vehicle 140.

The camera types for the cameras may include, but are not limited to,digital cameras that may be adapted for use with the components and/orsystems of the vehicle 140. The camera(s) may operate at automotivesafety integrity level (ASIL) B and/or at another ASIL. The camera typesmay be capable of any image capture rate, such as 60 frames per second(fps), 820 fps, 240 fps, etc., depending on the embodiment. The camerasmay be capable of using rolling shutters, global shutters, another typeof shutter, or a combination thereof. In some examples, a color filterarray of the cameras may include a red clear clear clear (RCCC) colorfilter array, a red clear clear blue (RCCB) color filter array, a redblue green clear (RBGC) color filter array, a Foveon X3 color filterarray, a Bayer sensors (RGGB) color filter array, a monochrome sensorcolor filter array, and/or another type of color filter array. In someembodiments, clear pixel cameras, such as cameras with an RCCC, an RCCB,and/or an RBGC color filter array, may be used in an effort to increaselight sensitivity.

In some examples, one or more of the camera(s) may be used to performAdvanced Driver Assistance Systems (ADAS) functions (e.g., as part of aredundant or fail-safe design). For example, a Multi-Function MonoCamera may be installed to provide functions including lane departurewarning, traffic sign assist and intelligent headlamp control. Also, oneor more of the camera(s) (e.g., all of the cameras) may record andprovide image data (e.g., video) simultaneously.

One or more of the cameras may be mounted in a mounting assembly, suchas a custom designed (3-D printed) assembly, in order to cut out straylight and reflections from within the car (e.g., reflections from thedashboard reflected in the windshield mirrors) which may interfere withthe camera's image data capture abilities. With reference to wing-mirrormounting assemblies, the wing-mirror assemblies may be custom 3-Dprinted so that the camera mounting plate matches the shape of thewing-mirror. In some examples, the camera(s) may be integrated into thewing-mirror. For side-view cameras, the camera(s) may also be integratedwithin the four pillars at each corner of the cabin.

Cameras with a field of view that includes portions of the environmentin front of the vehicle 140 (e.g., front-facing cameras) may be used forsurround view, to help identify forward facing paths and obstacles, aswell aid in, with the help of one or more controllers 836 and/or controlSoCs, providing information critical to generating an occupancy gridand/or determining the preferred vehicle paths. Front-facing cameras maybe used to perform many of the same ADAS functions as LIDAR, includingemergency braking, pedestrian detection, and collision avoidance.Front-facing cameras may also be used for ADAS functions and systemsincluding Lane Departure Warnings (LDW), Autonomous Cruise Control(ACC), and/or other functions such as traffic sign recognition.

A variety of cameras may be used in a front-facing configuration,including, for example, a monocular camera platform that includes aComplementary Metal Oxide Semiconductor (CMOS) color imager. Anotherexample may be a wide-view camera(s) 870 that may be used to perceiveobjects coming into view from the periphery (e.g., pedestrians, crossingtraffic or bicycles). Although only one wide-view camera is illustratedin FIG. 8B, there may any number of wide-view cameras 870 on the vehicle140. In addition, long-range camera(s) 898 (e.g., a long-view stereocamera pair) may be used for depth-based object detection, especiallyfor objects for which a neural network has not yet been trained. Thelong-range camera(s) 898 may also be used for object detection andclassification, as well as basic object tracking.

One or more stereo cameras 868 may also be included in a front-facingconfiguration. The stereo camera(s) 868 may include an integratedcontrol unit comprising a scalable processing unit, which may provide aprogrammable logic, such as a Field-Programmable Gate Array (FPGA), anda multi-core micro-processor with an integrated Controller Area Network(CAN) or Ethernet interface on a single chip. Such a unit may be used togenerate a 3-D map of the vehicle's environment, including a distanceestimate for all the points in the image. An alternative stereocamera(s) 868 may include a compact stereo vision sensor(s) that mayinclude two camera lenses (one each on the left and right) and an imageprocessing chip that may measure the distance from the vehicle to thetarget object and use the generated information (e.g., metadata) toactivate the autonomous emergency braking and lane departure warningfunctions. Other types of stereo camera(s) 868 may be used in additionto, or alternatively from, those described herein.

Cameras with a field of view that includes portions of the environmentto the side of the vehicle 140 (e.g., side-view cameras) may be used forsurround view, providing information used to create and update theoccupancy grid, as well as to generate side impact collision warnings.For example, surround camera(s) 874 (e.g., four surround cameras 874 asillustrated in FIG. 8B) may be positioned to on the vehicle 140. Thesurround camera(s) 874 may include wide-view camera(s) 870, fisheyecamera(s), 360 degree camera(s), and/or the like. Four example, fourfisheye cameras may be positioned on the vehicle's front, rear, andsides. In an alternative arrangement, the vehicle may use three surroundcamera(s) 874 (e.g., left, right, and rear), and may leverage one ormore other camera(s) (e.g., a forward-facing camera) as a fourthsurround view camera.

Cameras with a field of view that include portions of the environment tothe rear of the vehicle 140 (e.g., rear-view cameras) may be used forpark assistance, surround view, rear collision warnings, and creatingand updating the occupancy grid. A wide variety of cameras may be usedincluding, but not limited to, cameras that are also suitable as afront-facing camera(s) (e.g., long-range and/or mid-range camera(s) 898,stereo camera(s) 868), infrared camera(s) 872, etc.), as describedherein.

FIG. 8C is a block diagram of an example system architecture for theexample autonomous vehicle 140 of FIG. 8A, in accordance with someembodiments of the present disclosure. It should be understood that thisand other arrangements described herein are set forth only as examples.Other arrangements and elements (e.g., machines, interfaces, functions,orders, groupings of functions, etc.) can be used in addition to orinstead of those shown, and some elements may be omitted altogether.Further, many of the elements described herein are functional entitiesthat may be implemented as discrete or distributed components or inconjunction with other components, and in any suitable combination andlocation. Various functions described herein as being performed byentities may be carried out by hardware, firmware, and/or software. Forinstance, various functions may be carried out by a processor executinginstructions stored in memory.

Each of the components, features, and systems of the vehicle 140 in FIG.8C are illustrated as being connected via bus 802. The bus 802 mayinclude a CAN data interface (alternatively referred to herein as a “CANbus”), Ethernet, FlexRay, and/or another type of bus. A CAN may be anetwork inside the vehicle 140 used to aid in control of variousfeatures and functionality of the vehicle 140, such as actuation ofbrakes, acceleration, braking, steering, windshield wipers, etc. A CANbus may be configured to have dozens or even hundreds of nodes, eachwith its own unique identifier (e.g., a CAN ID). The CAN bus can be readto find steering wheel angle, ground speed, engine revolutions perminute (RPMs), button positions, and/or other vehicle status indicators.The CAN bus may be ASIL B compliant.

Although a single line is used to represent the bus 802, this is notintended to be limiting. For example, there may be any number of busses802, which may include one or more CAN busses, one or more FlexRaybusses, one or more Ethernet busses, and/or one or more other types ofbusses using a different protocol. In some examples, two or more busses802 may be used to perform different functions, and/or may be used forredundancy. For example, a first bus 802 may be used for collisionavoidance functionality and a second bus 802 may be used for actuationcontrol. In any example, each bus 802 may communicate with any of thecomponents of the vehicle 140, and two or more busses 802 maycommunicate with the same components. In some examples, each SoC 804,each controller 836, and/or each computer within the vehicle may haveaccess to the same input data (e.g., inputs from sensors of the vehicle140), and may be connected to a common bus, such the CAN bus.

The vehicle 140 may include one or more controller(s) 836, such as thosedescribed herein with respect to FIG. 8A. The controller(s) 836 may beused for a variety of functions. The controller(s) 836 may be coupled toany of the various other components and systems of the vehicle 140, andmay be used for control of the vehicle 140, artificial intelligence ofthe vehicle 140, infotainment for the vehicle 140, and/or the like.

The SoC 804 may include Central Processing Unit(s) (CPU) 806, GraphicsProcessing Unit(s) (GPU) 808, processor(s) 810, cache(s) 812,accelerator(s) 814, data store(s) 816, and/or other components andfeatures not illustrated. The SoC(s) 804 may be used to control thevehicle 140 in a variety of platforms and systems. For example, theSoC(s) 804 may be combined in a system (e.g., the system of the vehicle140) with an HD map 822 which may obtain map refreshes and/or updatesvia a network interface 824 from one or more servers (e.g., server(s)878 of FIG. 8D).

The CPU(s) 806 may include a CPU cluster or CPU complex (alternativelyreferred to herein as a “CCPLEX”). The CPU(s) 806 may include multiplecores and/or L2 caches. For example, in some embodiments, the CPU(s) 806may include eight cores in a coherent multi-processor configuration. Insome embodiments, the CPU(s) 806 may include four dual-core clusterswhere each cluster has a dedicated L2 cache (e.g., a 2 MB L2 cache). TheCPU(s) 806 (e.g., the CCPLEX) may be configured to support simultaneouscluster operation enabling any combination of the clusters of the CPU(s)806 to be active at any given time.

The CPU(s) 806 may implement power management capabilities that includeone or more of the following features: individual hardware blocks may beclock-gated automatically when idle to save dynamic power; each coreclock may be gated when the core is not actively executing instructionsdue to execution of WFI/WFE instructions; each core may be independentlypower-gated; each core cluster can be independently clock-gated when allcores are clock-gated or power-gated; and/or each core cluster can beindependently power-gated when all cores are power-gated. The CPU(s) 806may further implement an enhanced algorithm for managing power states,where allowed power states and expected wakeup times are specified, andthe hardware/microcode determines the best power state to enter for thecore, cluster, and CCPLEX. The processing cores may support simplifiedpower state entry sequences in software with the work offloaded tomicrocode.

The GPU(s) 808 may include an integrated GPU (alternatively referred toherein as an “iGPU”). The GPU(s) 808 may be programmable and may beefficient for parallel workloads. The GPU(s) 808, in some examples, mayuse an enhanced tensor instruction set. The GPU(s) 808 may include oneor more streaming microprocessors, where each streaming microprocessormay include an L1 cache (e.g., an L1 cache with at least 96 KB storagecapacity), and two or more of the streaming microprocessors may share anL2 cache (e.g., an L2 cache with a 512 KB storage capacity). In someembodiments, the GPU(s) 808 may include at least eight streamingmicroprocessors. The GPU(s) 808 may use compute application programminginterface(s) (API(s)). In addition, the GPU(s) 808 may use one or moreparallel computing platforms and/or programming models (e.g., NVIDIA'sCUDA).

The GPU(s) 808 may be power-optimized for best performance in automotiveand embedded use cases. For example, the GPU(s) 808 may be fabricatedusing Fin Field-Effect Transistor (FinFET) technologies. However, thisis not intended to be limiting and the GPU(s) 808 may be fabricatedusing other semiconductor manufacturing processes. Each streamingmicroprocessor may incorporate a number of mixed-precision processingcores partitioned into multiple blocks. For example, and withoutlimitation, 64 PF32 cores and 32 PF64 cores may be partitioned into fourprocessing blocks. In such an example, each processing block may beallocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, twomixed-precision NVIDIA TENSOR COREs for deep learning matrix arithmetic,an L0 instruction cache, a warp scheduler, a dispatch unit, and/or a 64KB register file. In addition, the streaming microprocessors may includeindependent parallel integer and floating-point data paths to providefor efficient execution of workloads with a mix of computation andaddressing calculations. The streaming microprocessors may includeindependent thread scheduling capability to enable finer-grainsynchronization and cooperation between parallel threads. The streamingmicroprocessors may include a combined L1 data cache and shared memoryunit in order to improve performance while simplifying programming.

The GPU(s) 808 may include a high bandwidth memory (HBM) and/or a 16 GBHBM2 memory subsystem to provide, in some examples, about 900 GB/secondpeak memory bandwidth. In some examples, in addition to, oralternatively from, the HBM memory, a Synchronous Graphics Random-AccessMemory (SGRAM) may be used, such as a Graphics Double Data Rate typeFive synchronous random-access memory (GDDR5).

The GPU(s) 808 may include unified memory technology including accesscounters to allow for more accurate migration of memory pages to theprocessor that accesses them most frequently, thereby improvingefficiency for memory ranges shared between processors. In someexamples, Address Translation Services (ATS) support may be used toallow the GPU(s) 808 to access the CPU(s) 806 page tables directly. Insuch examples, when the GPU(s) 808 Memory Management Unit (MMU)experiences a miss, an address translation request may be transmitted tothe CPU(s) 806. In response, the CPU(s) 806 may look in its page tablesfor the virtual-to-physical mapping for the address and transmits thetranslation back to the GPU(s) 808. As such, unified memory technologymay allow a single unified virtual address space for memory of both theCPU(s) 806 and the GPU(s) 808, thereby simplifying the GPU(s) 808programming and porting of applications to the GPU(s) 808.

In addition, the GPU(s) 808 may include an access counter that may keeptrack of the frequency of access of the GPU(s) 808 to memory of otherprocessors. The access counter may help ensure that memory pages aremoved to the physical memory of the processor that is accessing thepages most frequently.

The SoC(s) 804 may include any number of cache(s) 812, including thosedescribed herein. For example, the cache(s) 812 may include an L3 cachethat is available to both the CPU(s) 806 and the GPU(s) 808 (e.g., thatis connected both the CPU(s) 806 and the GPU(s) 808). The cache(s) 812may include a write-back cache that can keep track of states of lines,such as by using a cache coherence protocol (e.g., MEI, MESI, MSI,etc.). The L3 cache may include 4 MB or more, depending on theembodiment, although smaller cache sizes may be used.

The SoC(s) 804 may include one or more accelerators 814 (e.g., hardwareaccelerators, software accelerators, or a combination thereof). Forexample, the SoC(s) 804 may include a hardware acceleration cluster thatmay include optimized hardware accelerators and/or large on-chip memory.The large on-chip memory (e.g., 4 MB of SRAM), may enable the hardwareacceleration cluster to accelerate neural networks and othercalculations. The hardware acceleration cluster may be used tocomplement the GPU(s) 808 and to off-load some of the tasks of theGPU(s) 808 (e.g., to free up more cycles of the GPU(s) 808 forperforming other tasks). As an example, the accelerator(s) 814 may beused for targeted workloads (e.g., perception, Convolutional NeuralNetworks (CNNs), etc.) that are stable enough to be amenable toacceleration. The term “CNN,” as used herein, may include all types ofCNNs, including Region-based or Regional Convolutional Neural Networks(RCNNs) and Fast RCNNs (e.g., as used for object detection).

The accelerator(s) 814 (e.g., the hardware acceleration cluster) mayinclude a Deep Learning Accelerator(s) (DLA). The DLA(s) may include oneor more Tensor Processing Units (TPUs) that may be configured to providean additional ten trillion operations per second for deep learningapplications and inferencing. The TPUs may be accelerators configuredto, and optimized for, performing image processing functions (e.g., forCNNs, RCNNs, etc.). The DLA(s) may further be optimized for a specificset of neural network types and floating point operations, as well asinferencing. The design of the DLA(s) may provide more performance permillimeter than a general-purpose GPU, and vastly exceeds theperformance of a CPU. The TPU(s) may perform several functions,including a single-instance convolution function, supporting, forexample, INT8, INT16, and FP16 data types for both features and weights,as well as post-processor functions.

The DLA(s) may quickly and efficiently execute neural networks,especially CNNs, on processed or unprocessed data for any of a varietyof functions, including, for example and without limitation: a CNN forobject identification and detection using data from camera sensors; aCNN for distance estimation using data from camera sensors; a CNN foremergency vehicle detection and identification and detection using datafrom microphones; a CNN for facial recognition and vehicle owneridentification using data from camera sensors; and/or a CNN for securityand/or safety related events.

The DLA(s) may perform any function of the GPU(s) 808, and by using aninference accelerator, for example, a designer may target either theDLA(s) or the GPU(s) 808 for any function. For example, the designer mayfocus processing of CNNs and floating point operations on the DLA(s) andleave other functions to the GPU(s) 808 and/or other accelerator(s) 814.

The accelerator(s) 814 (e.g., the hardware acceleration cluster) mayinclude a Programmable Vision Accelerator(s) (PVA), which mayalternatively be referred to herein as a computer vision accelerator.The PVA(s) may be designed and configured to accelerate computer visionalgorithms for the Advanced Driver Assistance Systems (ADAS), autonomousdriving, and/or Augmented Reality (AR) and/or Virtual Reality (VR)applications. The PVA(s) may provide a balance between performance andflexibility. For example, each PVA(s) may include, for example andwithout limitation, any number of Reduced Instruction Set Computer(RISC) cores, Direct Memory Access (DMA), and/or any number of vectorprocessors.

The RISC cores may interact with image sensors (e.g., the image sensorsof any of the cameras described herein), image signal processor(s),and/or the like. Each of the RISC cores may include any amount ofmemory. The RISC cores may use any of a number of protocols, dependingon the embodiment. In some examples, the RISC cores may execute aReal-Time Operating System (RTOS). The RISC cores may be implementedusing one or more integrated circuit devices, Application SpecificIntegrated Circuits (ASICs), and/or memory devices. For example, theRISC cores may include an instruction cache and/or a tightly coupledRAM.

The DMA may enable components of the PVA(s) to access the system memoryindependently of the CPU(s) 806. The DMA may support any number offeatures used to provide optimization to the PVA including, but notlimited to, supporting multi-dimensional addressing and/or circularaddressing. In some examples, the DMA may support up to six or moredimensions of addressing, which may include block width, block height,block depth, horizontal block stepping, vertical block stepping, and/ordepth stepping.

The vector processors may be programmable processors that may bedesigned to efficiently and flexibly execute programming for computervision algorithms and provide signal processing capabilities. In someexamples, the PVA may include a PVA core and two vector processingsubsystem partitions. The PVA core may include a processor subsystem,DMA engine(s) (e.g., two DMA engines), and/or other peripherals. Thevector processing subsystem may operate as the primary processing engineof the PVA, and may include a Vector Processing Unit (VPU), aninstruction cache, and/or Vector Memory (e.g., VMEM). A VPU core mayinclude a digital signal processor such as, for example, a SingleInstruction, Multiple Data (SIMD), Very Long Instruction Word (VLIW)digital signal processor. The combination of the SIMD and VLIW mayenhance throughput and speed.

Each of the vector processors may include an instruction cache and maybe coupled to dedicated memory. As a result, in some examples, each ofthe vector processors may be configured to execute independently of theother vector processors. In other examples, the vector processors thatare included in a particular PVA may be configured to employ dataparallelism. For example, in some embodiments, the plurality of vectorprocessors included in a single PVA may execute the same computer visionalgorithm, but on different regions of an image. In other examples, thevector processors included in a particular PVA may simultaneouslyexecute different computer vision algorithms, on the same image, or evenexecute different algorithms on sequential images or portions of animage. Among other things, any number of PVAs may be included in thehardware acceleration cluster and any number of vector processors may beincluded in each of the PVAs. In addition, the PVA(s) may includeadditional Error Correcting Code (ECC) memory, to enhance overall systemsafety.

The accelerator(s) 814 (e.g., the hardware acceleration cluster) mayinclude a computer vision network on-chip and SRAM, for providing ahigh-bandwidth, low latency SRAM for the accelerator(s) 814. In someexamples, the on-chip memory may include at least 4 MB SRAM, consistingof, for example and without limitation, eight field-configurable memoryblocks, that may be accessible by both the PVA and the DLA. Each pair ofmemory blocks may include an Advanced Peripheral Bus (APB) interface,configuration circuitry, a controller, and a multiplexer. Any type ofmemory may be used. The PVA and DLA may access the memory via a backbonethat provides the PVA and DLA with high-speed access to memory. Thebackbone may include a computer vision network on-chip thatinterconnects the PVA and the DLA to the memory (e.g., using the APB).

The computer vision network on-chip may include an interface thatdetermines, before transmission of any control signal/address/data, thatboth the PVA and the DLA provide ready and valid signals. Such aninterface may provide for separate phases and separate channels fortransmitting control signals/addresses/data, as well as burst-typecommunications for continuous data transfer. This type of interface maycomply with ISO 26262 or IEC 68508 standards, although other standardsand protocols may be used.

In some examples, the SoC(s) 804 may include a real-time ray-tracinghardware accelerator, such as described in U.S. patent application Ser.No. 16/101,832, filed on Aug. 10, 2018. The real-time ray-tracinghardware accelerator may be used to quickly and efficiently determinethe positions and extents of objects (e.g., within a world model), togenerate real0time visualization simulations, for RADAR signalinterpretation, for sound propagation synthesis and/or analysis, forsimulation of SONAR systems, for general wave propagation simulation,for comparison to LIDAR data for purposes of localization and/or otherfunctions, and/or for other uses.

The accelerator(s) 814 (e.g., the hardware accelerator cluster) have awide array of uses for autonomous driving. The PVA may be used for keyprocessing stages in ADAS and autonomous vehicles. The PVA'scapabilities may be a good match for algorithmic domains needingpredictable processing, at low power and low latency. In other words,the PVA may perform well on semi-dense or dense regular computation,even on small data sets, which need predictable run-times with lowlatency and low power. Thus, in the context of platforms for autonomousvehicles, the PVAs may be designed to run classic computer visionalgorithms, as they are typically efficient at object detection andoperating on integer math.

For example, according to one embodiment of the technology, the PVA isused to perform computer stereo vision. A semi-global matching-basedalgorithm may be used in some examples, although this is not intended tobe limiting. Many applications for Level 3-5 autonomous driving requiremotion estimation/stereo matching on-the-fly (e.g., structure frommotion, pedestrian recognition, lane detection, etc.). The PVA mayperform computer stereo vision function on inputs from two monocularcameras.

In some examples, the PVA may be used to perform dense optical flow.According to process raw RADAR data (e.g., using a 4D Fast FourierTransform) to provide Processed RADAR. In other examples, the PVA isused for time of flight depth processing, by processing raw time offlight data to provide processed time of flight data, for example.

The DLA may be used to run any type of network to enhance control anddriving safety, including for example, a neural network that outputs ameasure of confidence for each object detection. Such a confidence valuemay be interpreted as a probability, or as providing a relative “weight”of each detection compared to other detections. This confidence valuemay be used by the system to make further decisions regarding whichdetections should be considered as true positive detections rather thanfalse positive detections. For example, the system can set a thresholdvalue for the confidence and consider only the detections exceeding thethreshold value as true positive detections. In an Automatic EmergencyBraking (AEB) system, false positive detections would cause the vehicleto automatically perform emergency braking, which is obviouslyundesirable. Therefore, only the most confident detections should beconsidered as triggers for AEB. The DLA can run a neural network forregressing the confidence value. The neural network can take as itsinput at least some subset of parameters, such as bounding boxdimensions, ground plane estimate obtained (e.g. from anothersubsystem), Inertial Measurement Unit (IMU) sensor 866 output thatcorrelates with the vehicle 140 orientation, distance, 3D locationestimates of the object obtained from the neural network and/or othersensors (e.g., LIDAR sensor(s) 864 or RADAR sensor(s) 860), amongothers.

The SoC(s) 804 may include data store(s) 816 (e.g., memory). The datastore(s) 816 may be on-chip memory of the SoC(s) 804, which may storeneural networks to be executed on the GPU and/or the DLA. In someexamples, the data store(s) 816 may be large enough in capacity to storemultiple instances of neural networks for redundancy and safety. Thedata store(s) 812 may comprise L2 or L3 cache(s) 812. Reference to thedata store(s) 816 may include reference to the memory associated withthe PVA, DLA, and/or other accelerator(s) 814, as described herein.

The SoC(s) 804 may include one or more processor(s) 810 (e.g., embeddedprocessors). The processor(s) 810 may include a boot and powermanagement processor that may be a dedicated processor and subsystem tohandle boot power and management functions and related securityenforcement. The boot and power management processor may be a part ofthe SoC(s) 804 boot sequence and may provide runtime power managementservices. The boot power and management processor may provide clock andvoltage programming, assistance in system low power state transitions,management of SoC(s) 804 thermals and temperature sensors, and/ormanagement of the SoC(s) 804 power states. Each temperature sensor maybe implemented as a ring-oscillator whose output frequency isproportional to temperature, and the SoC(s) 804 may use thering-oscillators to detect temperatures of the CPU(s) 806, GPU(s) 808,and/or accelerator(s) 814. If temperatures are determined to exceed athreshold, the boot and power management processor may enter atemperature fault routine and put the SoC(s) 804 into a lower powerstate and/or put the vehicle 140 into a chauffeur to safe stop mode(e.g., bring the vehicle 140 to a safe stop).

The processor(s) 810 may further include a set of embedded processorsthat may serve as an audio processing engine. The audio processingengine may be an audio subsystem that enables full hardware support formulti-channel audio over multiple interfaces, and a broad and flexiblerange of audio I/O interfaces. In some examples, the audio processingengine is a dedicated processor core with a digital signal processorwith dedicated RAM.

The processor(s) 810 may further include an always-on processor enginethat may provide necessary hardware features to support low power sensormanagement and wake use cases. The always-on processor engine mayinclude a processor core, a tightly coupled RAM, supporting peripherals(e.g., timers and interrupt controllers), various I/O controllerperipherals, and routing logic.

The processor(s) 810 may further include a safety cluster engine thatincludes a dedicated processor subsystem to handle safety management forautomotive applications. The safety cluster engine may include two ormore processor cores, a tightly coupled RAM, support peripherals (e.g.,timers, an interrupt controller, etc.), and/or routing logic. In asafety mode, the two or more cores may operate in a lockstep mode andfunction as a single core with comparison logic to detect anydifferences between their operations.

The processor(s) 810 may further include a real-time camera engine thatmay include a dedicated processor subsystem for handling real-timecamera management.

The processor(s) 810 may further include a high-dynamic range signalprocessor that may include an image signal processor that is a hardwareengine that is part of the camera processing pipeline.

The processor(s) 810 may include a video image compositor that may be aprocessing block (e.g., implemented on a microprocessor) that implementsvideo post-processing functions needed by a video playback applicationto produce the final image for the player window. The video imagecompositor may perform lens distortion correction on wide-view camera(s)870, surround camera(s) 874, and/or on in-cabin monitoring camerasensors. The in-cabin monitoring camera sensors may be monitored by aneural network running on another instance of the Advanced SoC,configured to identify in-cabin events and respond accordingly. Anin-cabin system may perform lip reading to activate cellular service andplace a phone call, dictate emails, change the vehicle's destination,activate or change the vehicle's infotainment system and settings, orprovide voice-activated web surfing. Certain functions are available tothe driver only when the vehicle is operating in an autonomous mode, andare disabled otherwise.

The video image compositor may include enhanced temporal noise reductionfor both spatial and temporal noise reduction. For example, where motionoccurs in a video, the noise reduction may weight spatial informationaccordingly, decreasing the weight of information provided by adjacentframes. Where an image or portion of an image does not include motion,the temporal noise reduction performed by the video image compositor mayuse information from the previous image to reduce noise in the currentimage.

The video image compositor may also be configured to perform stereorectification on input stereo lens frames. The video image compositormay further be used for user interface composition when the operatingsystem desktop is in use, and the GPU(s) 808 is not required tocontinuously render new surfaces. Even when the GPU(s) 808 is powered onand active doing 3D rendering, the video image compositor may be used tooffload the GPU(s) 808 to improve performance and responsiveness.

The SoC(s) 804 may further include a Mobile Industry Processor Interface(MIPI) camera serial interface for receiving video and input fromcameras, a high-speed interface, and/or a video input block that may beused for camera and related pixel input functions. The SoC(s) 804 mayfurther include an input/output controller(s) that may be controlled bysoftware and may be used for receiving I/O signals that are uncommittedto a specific role.

The SoC(s) 804 may further include a broad range of peripheralinterfaces to enable communication with peripherals, audio codecs, powermanagement, and/or other devices. The SoC(s) 804 may be used to processdata from cameras (e.g., connected over Gigabit Multimedia Serial Linkand Ethernet), sensors (e.g., LIDAR sensor(s) 864, RADAR sensor(s) 860,etc. that may be connected over Ethernet), data from bus 802 (e.g.,speed of vehicle 140, steering wheel position, etc.), data from GNSSsensor(s) 858 (e.g., connected over Ethernet or CAN bus). The SoC(s) 804may further include dedicated high-performance mass storage controllersthat may include their own DMA engines, and that may be used to free theCPU(s) 806 from routine data management tasks.

The SoC(s) 804 may be an end-to-end platform with a flexiblearchitecture that spans automation levels 3-5, thereby providing acomprehensive functional safety architecture that leverages and makesefficient use of computer vision and ADAS techniques for diversity andredundancy, provides a platform for a flexible, reliable drivingsoftware stack, along with deep learning tools. The SoC(s) 804 may befaster, more reliable, and even more energy-efficient andspace-efficient than conventional systems. For example, theaccelerator(s) 814, when combined with the CPU(s) 806, the GPU(s) 808,and the data store(s) 816, may provide for a fast, efficient platformfor level 3-5 autonomous vehicles.

The technology described herein may provide capabilities andfunctionality that cannot be achieved by conventional systems. Forexample, computer vision algorithms can be executed on CPUs, which canbe configured using high-level programming language, such as the Cprogramming language, to execute a wide variety of processing algorithmsacross a wide variety of visual data. However, CPUs are oftentimesunable to meet the performance requirements of many computer visionapplications, such as those related to execution time and powerconsumption, for example. In particular, many CPUs are unable to executecomplex object detection algorithms in real-time, which may be requiredfor in-vehicle ADAS applications, and for practical Level 3-5 autonomousvehicles.

By providing a CPU complex, GPU complex, and a hardware accelerationcluster, multiple neural networks may be used simultaneously and/orsequentially, and the results may be combined together to enable Level3-5 autonomous driving functionality. For example, a CNN executing onthe DLA or dGPU (e.g., the GPU(s) 820) may include a text and wordrecognition, allowing the supercomputer to read and understand trafficsigns, including signs for which the neural network has not beenspecifically trained. The DLA may further include a neural network thatis able to identify, interpret, and provides semantic understanding ofthe sign, and to pass that semantic understanding to the path planningmodules running on the CPU Complex.

As another example, multiple neural networks may be run simultaneously,as may be required for Level 3, 4, or 5 driving. For example, a warningsign consisting of “Caution: flashing lights indicate icy conditions,”along with an electric light, may be independently or collectivelyinterpreted by several neural networks. The sign itself may beidentified as a traffic sign by a first deployed neural network (e.g., aneural network that has been trained), the text “Flashing lightsindicate icy conditions” may be interpreted by a second deployed neuralnetwork, which informs the vehicle's path planning software (e.g.,executing on the CPU Complex) that when flashing lights are detected,icy conditions exist. The flashing light may be identified by operatinga third deployed neural network over multiple frames, informing thevehicle's path-planning software of the presence (or absence) offlashing lights. All three neural networks may run simultaneously, suchas within the DLA and/or on the GPU(s) 808.

In some examples, a CNN for facial recognition and vehicle owneridentification may use data from camera sensors to identify the presenceof an authorized driver and/or owner of the vehicle 140. The always onsensor processing engine may be used to unlock the vehicle when theowner approaches the driver door and turn on the lights, and, insecurity mode, to disable the vehicle when the owner leaves the vehicle.In this way, the SoC(s) 804 provide for security against theft and/orcarjacking.

In another example, a CNN for emergency vehicle detection andidentification may use data from microphones 896 to detect and identifyemergency vehicle sirens. In contrast to conventional systems, that usegeneral classifiers to detect sirens and manually extract features, theSoC(s) 804 use the CNN for classifying environmental and urban sounds,as well as classifying visual data. In some examples, the CNN running onthe DLA is trained to identify the relative closing speed of theemergency vehicle (e.g., by using the Doppler Effect). The CNN can alsobe trained to identify emergency vehicles specific to the local area inwhich the vehicle is operating, as identified by GNSS sensor(s) 858.Thus, for example, when operating in Europe the CNN will seek to detectEuropean sirens, and when in the United States the CNN will seek toidentify only North American sirens. Once an emergency vehicle isdetected, a control program may be used to execute an emergency vehiclesafety routine, slowing the vehicle, pulling over to the side of theroad, parking the vehicle, and/or idling the vehicle, with theassistance of ultrasonic sensors 862, until the emergency vehicle(s)passes.

The vehicle may include a CPU(s) 818 (e.g., discrete CPU(s), ordCPU(s)), that may be coupled to the SoC(s) 804 via a high-speedinterconnect (e.g., PCIe). The CPU(s) 818 may include an X86 processor,for example. The CPU(s) 818 may be used to perform any of a variety offunctions, including arbitrating potentially inconsistent resultsbetween ADAS sensors and the SoC(s) 804, and/or monitoring the statusand health of the controller(s) 836 and/or infotainment SoC 830, forexample.

The vehicle 140 may include a GPU(s) 820 (e.g., discrete GPU(s), ordGPU(s)), that may be coupled to the SoC(s) 804 via a high-speedinterconnect (e.g., NVIDIA's NVLINK). The GPU(s) 820 may provideadditional artificial intelligence functionality, such as by executingredundant and/or different neural networks, and may be used to trainand/or update neural networks based on input (e.g., sensor data) fromsensors of the vehicle 140.

The vehicle 140 may further include the network interface 824 which mayinclude one or more wireless antennas 826 (e.g., one or more wirelessantennas for different communication protocols, such as a cellularantenna, a Bluetooth antenna, etc.). The network interface 824 may beused to enable wireless connectivity over the Internet with the cloud(e.g., with the server(s) 878 and/or other network devices), with othervehicles, and/or with computing devices (e.g., client devices ofpassengers). To communicate with other vehicles, a direct link may beestablished between the two vehicles and/or an indirect link may beestablished (e.g., across networks and over the Internet). Direct linksmay be provided using a vehicle-to-vehicle communication link. Thevehicle-to-vehicle communication link may provide the vehicle 140information about vehicles in proximity to the vehicle 140 (e.g.,vehicles in front of, on the side of, and/or behind the vehicle 140).This functionality may be part of a cooperative adaptive cruise controlfunctionality of the vehicle 140.

The network interface 824 may include a SoC that provides modulation anddemodulation functionality and enables the controller(s) 836 tocommunicate over wireless networks. The network interface 824 mayinclude a radio frequency front-end for up-conversion from baseband toradio frequency, and down conversion from radio frequency to baseband.The frequency conversions may be performed through well-known processes,and/or may be performed using super-heterodyne processes. In someexamples, the radio frequency front end functionality may be provided bya separate chip. The network interface may include wirelessfunctionality for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000,Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and/or otherwireless protocols.

The vehicle 140 may further include data store(s) 828 which may includeoff-chip (e.g., off the SoC(s) 804) storage. The data store(s) 828 mayinclude one or more storage elements including RAM, SRAM, DRAM, VRAM,Flash, hard disks, and/or other components and/or devices that can storeat least one bit of data.

The vehicle 140 may further include GNSS sensor(s) 858. The GNSSsensor(s) 858 (e.g., GPS and/or assisted GPS sensors), to assist inmapping, perception, occupancy grid generation, and/or path planningfunctions. Any number of GNSS sensor(s) 858 may be used, including, forexample and without limitation, a GPS using a Universal Serial Bus (USB)connector with an Ethernet to Serial (RS-232) bridge.

The vehicle 140 may further include RADAR sensor(s) 860. The RADARsensor(s) 860 may be used by the vehicle 140 for long-range vehicledetection, even in darkness and/or severe weather conditions. RADARfunctional safety levels may be ASIL B. The RADAR sensor(s) 860 may usethe CAN and/or the bus 802 (e.g., to transmit data generated by theRADAR sensor(s) 860) for control and to access object tracking data,with access to Ethernet to access raw data in some examples. A widevariety of RADAR sensor types may be used. For example, and withoutlimitation, the RADAR sensor(s) 860 may be suitable for front, rear, andside RADAR use. In some example, Pulse Doppler RADAR sensor(s) are used.

The RADAR sensor(s) 860 may include different configurations, such aslong range with narrow field of view, short range with wide field ofview, short range side coverage, etc. In some examples, long-range RADARmay be used for adaptive cruise control functionality. The long-rangeRADAR systems may provide a broad field of view realized by two or moreindependent scans, such as within a 250 m range. The RADAR sensor(s) 860may help in distinguishing between static and moving objects, and may beused by ADAS systems for emergency brake assist and forward collisionwarning. Long-range RADAR sensors may include monostatic multimodalRADAR with multiple (e.g., six or more) fixed RADAR antennae and ahigh-speed CAN and FlexRay interface. In an example with six antennae,the central four antennae may create a focused beam pattern, designed torecord the vehicle's 140 surroundings at higher speeds with minimalinterference from traffic in adjacent lanes. The other two antennae mayexpand the field of view, making it possible to quickly detect vehiclesentering or leaving the vehicle's 140 lane.

Mid-range RADAR systems may include, as an example, a range of up to 860m (front) or 80 m (rear), and a field of view of up to 42 degrees(front) or 850 degrees (rear). Short-range RADAR systems may include,without limitation, RADAR sensors designed to be installed at both endsof the rear bumper. When installed at both ends of the rear bumper, sucha RADAR sensor systems may create two beams that constantly monitor theblind spot in the rear and next to the vehicle.

Short-range RADAR systems may be used in an ADAS system for blind spotdetection and/or lane change assist.

The vehicle 140 may further include ultrasonic sensor(s) 862. Theultrasonic sensor(s) 862, which may be positioned at the front, back,and/or the sides of the vehicle 140, may be used for park assist and/orto create and update an occupancy grid. A wide variety of ultrasonicsensor(s) 862 may be used, and different ultrasonic sensor(s) 862 may beused for different ranges of detection (e.g., 2.5 m, 4 m). Theultrasonic sensor(s) 862 may operate at functional safety levels of ASILB.

The vehicle 140 may include LIDAR sensor(s) 864. The LIDAR sensor(s) 864may be used for object and pedestrian detection, emergency braking,collision avoidance, and/or other functions. The LIDAR sensor(s) 864 maybe functional safety level ASIL B. In some examples, the vehicle 140 mayinclude multiple LIDAR sensors 864 (e.g., two, four, six, etc.) that mayuse Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

In some examples, the LIDAR sensor(s) 864 may be capable of providing alist of objects and their distances for a 360-degree field of view.Commercially available LIDAR sensor(s) 864 may have an advertised rangeof approximately 140 m, with an accuracy of 2 cm-3 cm, and with supportfor a 140 Mbps Ethernet connection, for example. In some examples, oneor more non-protruding LIDAR sensors 864 may be used. In such examples,the LIDAR sensor(s) 864 may be implemented as a small device that may beembedded into the front, rear, sides, and/or corners of the vehicle 140.The LIDAR sensor(s) 864, in such examples, may provide up to an820-degree horizontal and 35-degree vertical field-of-view, with a 200 mrange even for low-reflectivity objects. Front-mounted LIDAR sensor(s)864 may be configured for a horizontal field of view between 45 degreesand 135 degrees.

In some examples, LIDAR technologies, such as 3D flash LIDAR, may alsobe used. 3D Flash LIDAR uses a flash of a laser as a transmissionsource, to illuminate vehicle surroundings up to approximately 200 m. Aflash LIDAR unit includes a receptor, which records the laser pulsetransit time and the reflected light on each pixel, which in turncorresponds to the range from the vehicle to the objects. Flash LIDARmay allow for highly accurate and distortion-free images of thesurroundings to be generated with every laser flash. In some examples,four flash LIDAR sensors may be deployed, one at each side of thevehicle 140. Available 3D flash LIDAR systems include a solid-state 3Dstaring array LIDAR camera with no moving parts other than a fan (e.g.,a non-scanning LIDAR device). The flash LIDAR device may use a 5nanosecond class I (eye-safe) laser pulse per frame and may capture thereflected laser light in the form of 3D range point clouds andco-registered intensity data. By using flash LIDAR, and because flashLIDAR is a solid-state device with no moving parts, the LIDAR sensor(s)864 may have low susceptibility to motion blur, vibration, and/or shock.

The vehicle may further include IMU sensor(s) 866. The IMU sensor(s) 866may be located at a center of the rear axle of the vehicle 140, in someexamples. The IMU sensor(s) 866 may include, for example and withoutlimitation, an accelerometer(s), a magnetometer(s), a gyroscope(s), amagnetic compass(es), and/or other sensor types. In some examples, suchas in six-axis applications, the IMU sensor(s) 866 may includeaccelerometers and gyroscopes, while in nine-axis applications, the IMUsensor(s) 866 may include accelerometers, gyroscopes, and magnetometers.

In some embodiments, the IMU sensor(s) 866 may be implemented as aminiature, high performance GPS-Aided Inertial Navigation System(GPS/INS) that combines micro-electro-mechanical systems (MEMS) inertialsensors, a high-sensitivity GPS receiver, and advanced Kalman filteringalgorithms to provide estimates of position, velocity, and altitude. Assuch, in some examples, the IMU sensor(s) 866 may enable the vehicle 140to estimate heading without requiring input from a magnetic sensor bydirectly observing and correlating the changes in velocity from GPS tothe IMU sensor(s) 866. In some examples, the IMU sensor(s) 866 and theGNSS sensor(s) 858 may be combined in a single integrated unit.

The vehicle may include microphone(s) 896 placed in and/or around thevehicle 140. The microphone(s) 896 may be used for emergency vehicledetection and identification, among other things.

The vehicle may further include any number of camera types, includingstereo camera(s) 868, wide-view camera(s) 870, infrared camera(s) 872,surround camera(s) 874, long-range and/or mid-range camera(s) 898,and/or other camera types. The cameras may be used to capture image dataaround an entire periphery of the vehicle 140. The types of cameras useddepends on the embodiments and requirements for the vehicle 140, and anycombination of camera types may be used to provide the desired coveragearound the vehicle 140. In addition, the number of cameras may differdepending on the embodiment. For example, the vehicle may include sixcameras, seven cameras, ten cameras, twelve cameras, and/or anothernumber of cameras. The cameras may support, as an example and withoutlimitation, Gigabit Multimedia Serial Link (GMSL) and/or GigabitEthernet. Each of the camera(s) is described with more detail hereinwith respect to FIG. 8A and FIG. 8B.

The vehicle 140 may further include vibration sensor(s) 842. Thevibration sensor(s) 842 may measure vibrations of components of thevehicle, such as the axle(s). For example, changes in vibrations mayindicate a change in road surfaces. In another example, when two or morevibration sensors 842 are used, the differences between the vibrationsmay be used to determine friction or slippage of the road surface (e.g.,when the difference in vibration is between a power-driven axle and afreely rotating axle).

The vehicle 140 may include an ADAS system 838. The ADAS system 838 mayinclude a SoC, in some examples. The ADAS system 838 may includeAutonomous/Adaptive/Automatic Cruise Control (ACC), Cooperative AdaptiveCruise Control (CACC), Forward Crash Warning (FCW), Automatic EmergencyBraking (AEB), Lane Departure Warnings (LDW), Lane Keep Assist (LKA),Blind Spot Warning (BSW), Rear Cross-Traffic Warning (RCTW), CollisionWarning Systems (CWS), Lane Centering (LC), and/or other features andfunctionality.

The ACC systems may use RADAR sensor(s) 860, LIDAR sensor(s) 864, and/ora camera(s). The ACC systems may include longitudinal ACC and/or lateralACC. Longitudinal ACC monitors and controls the distance to the vehicleimmediately ahead of the vehicle 140 and automatically adjust thevehicle speed to maintain a safe distance from vehicles ahead. LateralACC performs distance keeping, and advises the vehicle 140 to changelanes when necessary. Lateral ACC is related to other ADAS applicationssuch as Lane Change Assistance (LCA) and Collision Warning System (CWS).

CACC uses information from other vehicles that may be received via thenetwork interface 824 and/or the wireless antenna(s) 826 from othervehicles via a wireless link, or indirectly, over a network connection(e.g., over the Internet). Direct links may be provided by aVehicle-to-Vehicle (V2V) communication link, while indirect links may beInfrastructure-to-Vehicle (I2V) communication link. In general, the V2Vcommunication concept may provide information about the immediatelypreceding vehicles (e.g., vehicles immediately ahead of and in the samelane as the vehicle 140), while the I2V communication concept mayprovide information about traffic further ahead. CACC systems caninclude either or both I2V and V2V information sources. Given theinformation of the vehicles ahead of the vehicle 140, CACC can be morereliable with the potential to improve traffic flow smoothness andreduce congestion on the road.

FCW systems may be designed to alert the driver to a hazard, so that thedriver can take corrective action. FCW systems may use a front-facingcamera and/or RADAR sensor(s) 860, coupled to a dedicated processor,DSP, FPGA, and/or ASIC, that is electrically coupled to driver feedback,such as a display, speaker, and/or vibrating component. FCW systems mayprovide a warning, such as in the form of a sound, visual warning,vibration and/or a quick brake pulse.

AEB systems may detect an impending forward collision with anothervehicle or other object, and may automatically apply the brakes, such asif the driver does not take corrective action within a specified time ordistance parameter. AEB systems may use front-facing camera(s) and/orRADAR sensor(s) 860, coupled to a dedicated processor, DSP, FPGA, and/orASIC. When the AEB system detects a hazard, it may first alert thedriver to take corrective action to avoid the collision and, if thedriver does not take corrective action, the AEB system may automaticallyapply the brakes in an effort to prevent, or at least mitigate, theimpact of the predicted collision. AEB systems, may include techniquessuch as dynamic brake support and/or crash imminent braking.

LDW systems may provide visual, audible, and/or tactile warnings, suchas steering wheel or seat vibrations, to alert the driver when thevehicle 140 crosses lane markings. An LDW system may not activate whenthe driver indicates an intentional lane departure, such as byactivating a turn signal. LDW systems may use front-side facing cameras,coupled to a dedicated processor, DSP, FPGA, and/or ASIC, that iselectrically coupled to driver feedback, such as a display, speaker,and/or vibrating component.

LKA systems may be a variation of LDW systems. LKA systems may providesteering input or braking to correct the vehicle 140 if the vehicle 140starts to exit the lane.

BSW systems may detect and warn the driver of vehicles in anautomobile's blind spot. BSW systems may provide a visual, audible,and/or tactile alert to indicate that merging or changing lanes isunsafe. The system may provide an additional warning when the driveruses a turn signal. BSW systems may use rear-side facing camera(s)and/or RADAR sensor(s) 860, coupled to a dedicated processor, DSP, FPGA,and/or ASIC, that is electrically coupled to driver feedback, such as adisplay, speaker, and/or vibrating component.

RCTW systems may provide a visual, audible, and/or tactile notificationwhen an object is detected outside the rear-camera range when thevehicle 140 is backing up. Some RCTW systems include AEB to ensure thatthe vehicle brakes are applied to avoid a crash. RCTW systems may useone or more rear-facing RADAR sensor(s) 860, coupled to a dedicatedprocessor, DSP, FPGA, and/or ASIC, that is electrically coupled todriver feedback, such as a display, speaker, and/or vibrating component.

Some ADAS systems may be prone to false positive results which may beannoying and distracting to a driver, but typically are notcatastrophic, because the ADAS systems alert the driver and allow thedriver to decide whether a safety condition truly exists and actaccordingly. However, in an autonomous vehicle 140, the vehicle 140itself may, in the case of conflicting results, decide whether to heedthe result from a primary computer or a secondary computer (e.g., afirst controller 836 or a second controller 836). For example, in somecases, the ADAS system 838 may be a backup and/or secondary computer forproviding perception information to a backup computer rationalitymodule. The backup computer rationality monitor may run a redundantdiverse software on hardware components to detect faults in perceptionand dynamic driving tasks. Outputs from the ADAS system 838 may beprovided to a supervisory Multipoint Control Unit (MCU). If outputs fromthe primary computer and the secondary computer conflict, thesupervisory MCU may determine how to reconcile the conflict to ensuresafe operation.

In some examples, the primary computer may be configured to provide thesupervisory MCU with a confidence score, indicating the primarycomputer's confidence in the chosen result. If the confidence scoreexceeds a threshold, the supervisory MCU may follow the primarycomputer's direction, regardless of whether the secondary computerprovides a conflicting or inconsistent result. Where the confidencescore does not meet the threshold, and where the primary and secondarycomputer indicate different results (e.g., the conflict), thesupervisory MCU may arbitrate between the computers to determine theappropriate outcome.

The supervisory MCU may be configured to run a neural network(s) that istrained and configured to determine, based on outputs from the primarycomputer and the secondary computer, conditions under which thesecondary computer provides false alarms. Thus, the neural network(s) inthe supervisory MCU may learn when the secondary computer's output canbe trusted, and when it cannot. For example, when the secondary computeris a RADAR-based FCW system, a neural network(s) in the supervisory MCUmay learn when the FCW system is identifying metallic objects that arenot, in fact, hazards, such as a drainage grate or manhole cover thattriggers an alarm. Similarly, when the secondary computer is acamera-based LDW system, a neural network in the supervisory MCU maylearn to override the LDW when bicyclists or pedestrians are present anda lane departure is, in fact, the safest maneuver. In embodiments thatinclude a neural network(s) running on the supervisory MCU, thesupervisory MCU may include at least one of a DLA or GPU suitable forrunning the neural network(s) with associated memory. In someembodiments, the supervisory MCU may comprise and/or be included as acomponent of the SoC(s) 804.

In other examples, ADAS system 838 may include a secondary computer thatperforms ADAS functionality using traditional rules of computer vision.As such, the secondary computer may use classic computer vision rules(if-then), and the presence of a neural network(s) in the supervisoryMCU may improve reliability, safety, and performance. For example, thediverse implementation and intentional non-identity may make the overallsystem more fault-tolerant, especially to faults caused by software (orsoftware-hardware interface) functionality. For example, if there is asoftware bug or error in the software running on the primary computer,and the non-identical software code running on the secondary computerprovides the same overall result, the supervisory MCU may have greaterconfidence that the overall result is correct, and the bug in softwareor hardware on primary computer is not causing material error.

In some examples, the output of the ADAS system 838 may be fed into theprimary computer's perception block and/or the primary computer'sdynamic driving task block. For example, if the ADAS system 838indicates a forward crash warning due to an object immediately ahead,the perception block may use this information when identifying objects.In other examples, the secondary computer may have its own neuralnetwork which is trained and thus reduces the risk of false positives,as described herein.

The vehicle 140 may further include the infotainment SoC 830 (e.g., anin-vehicle infotainment system (IVI)). Although illustrated anddescribed as a SoC, the infotainment system may not be a SoC, and mayinclude two or more discrete components. The infotainment SoC 830 mayinclude a combination of hardware and software that may be used toprovide audio (e.g., music, a personal digital assistant, navigationalinstructions, news, radio, etc.), video (e.g., TV, movies, streaming,etc.), phone (e.g., hands-free calling), network connectivity (e.g.,LTE, WiFi, etc.), and/or information services (e.g., navigation systems,rear-parking assistance, a radio data system, vehicle relatedinformation such as fuel level, total distance covered, brake fuellevel, oil level, door open/close, air filter information, etc.) to thevehicle 140. For example, the infotainment SoC 830 may include radios,disk players, navigation systems, video players, USB and Bluetoothconnectivity, carputers, in-car entertainment, WiFi, steering wheelaudio controls, hands free voice control, a heads-up display (HUD), anHMI display 834, a telematics device, a control panel (e.g., forcontrolling and/or interacting with various components, features, and/orsystems), and/or other components. The infotainment SoC 830 may furtherbe used to provide information (e.g., visual and/or audible) to auser(s) of the vehicle, such as information from the ADAS system 838,autonomous driving information such as planned vehicle maneuvers,trajectories, surrounding environment information (e.g., intersectioninformation, vehicle information, road information, etc.), and/or otherinformation.

The infotainment SoC 830 may include GPU functionality. The infotainmentSoC 830 may communicate over the bus 802 (e.g., CAN bus, Ethernet, etc.)with other devices, systems, and/or components of the vehicle 140. Insome examples, the infotainment SoC 830 may be coupled to a supervisoryMCU such that the GPU of the infotainment system may perform someself-driving functions in the event that the primary controller(s) 836(e.g., the primary and/or backup computers of the vehicle 140) fail. Insuch an example, the infotainment SoC 830 may put the vehicle 140 into achauffeur to safe stop mode, as described herein.

The vehicle 140 may further include an instrument cluster 832 (e.g., adigital dash, an electronic instrument cluster, a digital instrumentpanel, etc.). The instrument cluster 832 may include a controller and/orsupercomputer (e.g., a discrete controller or supercomputer). Theinstrument cluster 832 may include a set of instrumentation such as aspeedometer, fuel level, oil pressure, tachometer, odometer, turnindicators, gearshift position indicator, seat belt warning light(s),parking-brake warning light(s), engine-malfunction light(s), airbag(SRS) system information, lighting controls, safety system controls,navigation information, etc. In some examples, information may bedisplayed and/or shared among the infotainment SoC 830 and theinstrument cluster 832. In other words, the instrument cluster 832 maybe included as part of the infotainment SoC 830, or vice versa.

FIG. 8D is a system diagram for communication between cloud-basedserver(s) and the example autonomous vehicle 140 of FIG. 8A, inaccordance with some embodiments of the present disclosure. The system876 may include server(s) 878, network(s) 890, and vehicles, includingthe vehicle 140. The server(s) 878 may include a plurality of GPUs884(A)-884(H) (collectively referred to herein as GPUs 884), PCIeswitches 882(A)-882(H) (collectively referred to herein as PCIe switches882), and/or CPUs 880(A)-880(B) (collectively referred to herein as CPUs880). The GPUs 884, the CPUs 880, and the PCIe switches may beinterconnected with high-speed interconnects such as, for example andwithout limitation, NVLink interfaces 888 developed by NVIDIA and/orPCIe connections 886. In some examples, the GPUs 884 are connected viaNVLink and/or NVSwitch SoC and the GPUs 884 and the PCIe switches 882are connected via PCIe interconnects. Although eight GPUs 884, two CPUs880, and two PCIe switches are illustrated, this is not intended to belimiting. Depending on the embodiment, each of the server(s) 878 mayinclude any number of GPUs 884, CPUs 880, and/or PCIe switches. Forexample, the server(s) 878 may each include eight, sixteen, thirty-two,and/or more GPUs 884.

The server(s) 878 may receive, over the network(s) 890 and from thevehicles, image data representative of images showing unexpected orchanged road conditions, such as recently commenced road-work. Theserver(s) 878 may transmit, over the network(s) 890 and to the vehicles,neural networks 892, updated neural networks 892, and/or map information894, including information regarding traffic and road conditions. Theupdates to the map information 894 may include updates for the HD map822, such as information regarding construction sites, potholes,detours, flooding, and/or other obstructions. In some examples, theneural networks 892, the updated neural networks 892, and/or the mapinformation 894 may have resulted from new training and/or experiencesrepresented in data received from any number of vehicles in theenvironment, and/or based on training performed at a datacenter (e.g.,using the server(s) 878 and/or other servers).

The server(s) 878 may be used to train machine learning models (e.g.,neural networks) based on training data (e.g., any combination of themachine learning models described herein). The training data may begenerated by the vehicles, and/or may be generated in a simulation(e.g., using a game engine). In some examples, the training data istagged (e.g., where the neural network benefits from supervisedlearning) and/or undergoes other pre-processing, while in other examplesthe training data is not tagged and/or pre-processed (e.g., where theneural network does not require supervised learning). Once the machinelearning models are trained, the machine learning models may be used bythe vehicles (e.g., transmitted to the vehicles over the network(s) 890,and/or the machine learning models may be used by the server(s) 878 toremotely monitor and/or control the vehicles.

In some examples, the server(s) 878 may receive data from the vehiclesand apply the data to up-to-date real-time neural networks for real-timeintelligent inferencing. The server(s) 878 may include deep-learningsupercomputers and/or dedicated AI computers powered by GPU(s) 884, suchas a DGX and DGX Station machines developed by NVIDIA. However, in someexamples, the server(s) 878 may include deep learning infrastructurethat use only CPU-powered datacenters.

The deep-learning infrastructure of the server(s) 878 may be capable offast, real-time inferencing, and may use that capability to evaluate andverify the health of the processors, software, and/or associatedhardware in the vehicle 140. For example, the deep-learninginfrastructure may receive periodic updates from the vehicle 140, suchas a sequence of images and/or objects that the vehicle 140 has locatedin that sequence of images (e.g., via computer vision and/or othermachine learning object classification techniques). The deep-learninginfrastructure may run its own neural network to identify the objectsand compare them with the objects identified by the vehicle 140 and, ifthe results do not match and the infrastructure concludes that the AI inthe vehicle 140 is malfunctioning, the server(s) 878 may transmit asignal to the vehicle 140 instructing a fail-safe computer of thevehicle 140 to assume control, notify the passengers, and complete asafe parking maneuver.

For inferencing, the server(s) 878 may include the GPU(s) 884 and one ormore programmable inference accelerators (e.g., NVIDIA's TensorRT 3).The combination of GPU-powered servers and inference acceleration maymake real-time responsiveness possible. In other examples, such as whereperformance is less critical, servers powered by CPUs, FPGAs, and otherprocessors may be used for inferencing.

The disclosure may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program modules including routines, programs,objects, components, data structures, etc., refer to code that performparticular tasks or implement particular abstract data types. Thedisclosure may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The disclosure mayalso be practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

The elements (e.g., systems, components, features, machines, interfaces,functions, orders, groupings of functions, functionality, and/orarrangements) described with respect to embodiments of the presentdisclosure are set forth only as examples. Other elements other thanthose described herein may be used in addition to or instead of thosedescribed herein, and some elements may be omitted altogether. Further,many of the elements described herein are functional entities that maybe implemented as discrete or distributed components or in conjunctionwith other components, and in any suitable combination and location.Various functions described herein as being performed by entities may becarried out by hardware, firmware, and/or software. For instance,various functions may be carried out by a processor executinginstructions stored in memory.

As used herein, a recitation of “and/or” with respect to two or moreelements should be interpreted to mean only one element, or acombination of elements. For example, “element A, element B, and/orelement C” may include only element A, only element B, only element C,element A and element B, element A and element C, element B and elementC, or elements A, B, and C. In addition, “at least one of element A orelement B” may include at least one of element A, at least one ofelement B, or at least one of element A and at least one of element B.Further, “at least one of element A and element B” may include at leastone of element A, at least one of element B, or at least one of elementA and at least one of element B.

The subject matter of the present disclosure is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of thisdisclosure. Rather, the inventors have contemplated that the claimedsubject matter might also be embodied in other ways, to includedifferent steps or combinations of steps similar to the ones describedin this document, in conjunction with other present or futuretechnologies. Moreover, although the terms “step” and/or “block” may beused herein to connote different elements of methods employed, the termsshould not be interpreted as implying any particular order among orbetween various steps herein disclosed unless and except when the orderof individual steps is explicitly described.

What is claimed is:
 1. A processor comprising: processing circuitry to: compute, using a deep neural network (DNN) and based at least in part on sensor data generated using one or more sensors of an autonomous machine and map data generated using a map, one or more trajectory points corresponding to a recommended trajectory for navigating a portion of an environment represented by the map data; and performing one or more operations by the autonomous machine based at least in part on the one or more trajectory points.
 2. The processor of claim 1, wherein the one or more operations include at least one of a planning operation or a control operation.
 3. The processor of claim 1, wherein the map includes at least one of a high definition map, a global navigation satellite system (GNSS) map, or a two-dimensional map.
 4. The processor of claim 1, wherein the DNN includes one or more first streams of layers to process the sensor data, one or more second streams of layers to process the map data, and one or more layers to compute, based at least in part on outputs of the first stream of layers and the second stream of layers, output data used for the computation of the one or more trajectory points.
 5. The processor of claim 1, further comprising processing circuitry to: apply a value of a control input to the DNN in addition to the sensor data and the map data, the value of the control input being indicative of progress of the autonomous machine through a maneuver, wherein the computation of the one or more trajectory points is further based at least in part on the value of the control input.
 6. The processor of claim 1, wherein the map data is representative of a road layout and a proposed path through the road layout.
 7. The processor of claim 1, wherein the computation of the one or more trajectory points is executed responsive to receiving data representative of a trigger signal indicating a start of a turning maneuver.
 8. The processor of claim 1, wherein the recommended trajectory corresponds to at least one of a lane change, a turn, or a lane split.
 9. The processor of claim 1, wherein the processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using an edge device; or a system implemented at least partially using cloud computing resources.
 10. A system comprising: one or more sensors; one or more processing units; and one or more memory units storing instruction thereon that, when executed using the one or more processing units, cause the one or more processing units to execute operations comprising: computing, using a deep neural network (DNN) and based at least in part on sensor data generated using the one or more sensors and map data representative of an image generated using a map application, data representative of at least a portion of a recommended trajectory for traversing an environment represented by the image; and performing one or more operations by the autonomous machine based at least in part on at least the portion of the recommended trajectory.
 11. The system of claim 10, wherein: the map data is applied to a first stream of the DNN; the sensor data is applied to a second stream of the DNN; and a first output of the first stream and a second output of the second stream are combined at a layer of the DNN.
 12. The system of claim 10, wherein the map application includes a global navigation satellite system (GNSS) map application, and the image is of a screenshot of the GNSS map application.
 13. The system of claim 10, wherein the image includes a proposed path through the portion of a map represented by the image.
 14. The system of claim 10, wherein the computation of the data is executed responsive to receiving data representative of a trigger signal indicating a start of a turning maneuver.
 15. The system of claim 10, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using an edge device; or a system implemented at least partially using cloud computing resources.
 16. A method comprising: responsive to a trigger signal indicating a start of a turning maneuver, computing, using a deep neural network (DNN) and based at least in part on sensor data generated using one or more sensors of an autonomous machine and map data representative of a road layout, data representative of at least a portion of a recommended trajectory for the autonomous machine through at least a portion of the road layout; and performing one or more operations by the autonomous machine based at least in part on at least the portion of the recommended trajectory.
 17. The method of claim 16, wherein the data is indicative of trajectory points corresponding to the recommended trajectory.
 18. The method of claim 16, further comprising: applying a value of a control input to the DNN, the value of the control input being indicative of progress of the autonomous machine through the turning maneuver, wherein the computing the data is further based at least in part on the value of the control input.
 19. The method of claim 18, wherein, at each iteration of the DNN, the value of the control input is updated.
 20. The method of claim 16, wherein the map data is representative of an image generated using a map of a mapping application, the map including a high definition map, a global navigation satellite system (GNSS) map, or a two-dimensional map. 